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EXECUTIVE SUMMARY 


Executive Summary 


The Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided 
grants to community colleges and other eligible institutions of higher education to expand and improve 
their ability to deliver education and career training programs. Grant-funded colleges served workers who 
were eligible for training under the Trade Adjustment Assistance for Workers (TAA) program as well as 
other adults in need of training for in-demand occupations. In doing so, the grant program supported 
the U.S. Department of Labor’s (DOL) goal of training workers with the skills to succeed in fast-growing, 
high-wage occupations. Between 2011 and 2014, the grant program awarded $1.9 billion across four 
rounds of grants to more than 1,000 colleges nationwide.? The fourth and final round of grants 

(71 grantees, 263 colleges) ended in 2018. 


In order to build evidence on grant-funded programs and strategies, DOL funded a national evaluation of 
each grant round to collect and assess qualitative and quantitative data across all participating colleges. 
The Round 4 evaluation included an outcomes study, the focus of this report. 


THE OUTCOMES STUDY 


This report describes the training, employment, earnings, and self-sufficiency outcomes of nearly 
2,800 participants who enrolled in 34 programs at nine Round 4 grantees. The overarching research 
questions are: 


1. What were the characteristics of study participants? 


2. In which features of training programs and services did study participants engage? 


3. What education outcomes did study participants in short-term training programs 
achieve? 


4. Did participants obtain employment? If so, was it in an occupation related to their 
training? What were their earnings? What were the characteristics of their jobs? 


The programs were grant-funded, generally short-term, and culminated in an industry-recognized 
credential. Each program had a distinct entry point, which facilitated enrollment of study participants. 
Exhibit ES-1 shows the location of the selected grantees, which included four single-institution grantees 
and five consortium grantees with a lead college. 


1 Congress authorized the TAACCCT grant program as part of the American Recovery and Reinvestment Act of 2009 to 
increase the capacity of community colleges to meet local and regional labor demand for a skilled workforce. Funding for 
the TAACCCT program was provided by the Health Care and Education Reconciliation Act of 2010. In their applications, 
grantees needed to provide data and analysis of both current and projected employment opportunities for each targeted 
industry and specific occupation. This must include data on current and expected job openings with at least two 
employers in the community in each targeted industry, and may include commitments from employers who expect to 
hire program participants. 


2 According to Cohen et al. (2017, 3), “729 unduplicated colleges participated in TAACCCT projects.” However, the 
authors note 729 could underestimate the number of unduplicated colleges because some grantees classified members 
of their consortium as community college districts rather than individual community colleges. The number of duplicated 
colleges across rounds is 1,113. 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1 vi 


EXECUTIVE SUMMARY 


Exhibit ES-1. Round 4 Grantees in the Outcomes Study 


MN 
Manchester 
i Community College —@ 
South Central i 
College 
OH 
CA aM 
KS ai Cincinnati State 
Ivy Tech Technical and 
iaahienrent Community Community College 
: ; College 
University 
Chaffey College 
Bossier Parish _f 
Community College yi 
FL 
Delgado 
Community 
College 
Bl Single Institution © @ Consortium Miami Dade 


Grantees Grantees College 


The grantees and programs were diverse. 


e The occupation focus of the programs spanned six industries, the most common of which was 
manufacturing? (total of 14 programs across five grantees), followed by information technology (total 
of nine programs in one grantee), welding (total of five programs across five grantees), construction 
(total of three programs in one grantee), health care (total of two programs in one grantee), and 
transportation and warehousing (one program in one grantee). 


e Programs ranged in duration from one week to 24 months. The mean program duration was 
8.7 months, with a median of 6.5. 


— Manufacturing programs ranged from 1 to 16 months. 

— Information Technology programs ranged from 4 to 24 months. 
— Welding programs ranged from 4 to 18 months. 

— Construction programs ranged from 3 to 6 months; 

— Health Care programs ranged from 5 to 8 months; 


—  Transportation/Warehousing program was 1 week to 1 month. 


3 Manufacturing includes advanced manufacturing. As noted in an earlier footnote, the grantees determined which 


occupations were in demand using labor market information. They had to demonstrate the demand for certain 
occupations in their grant applications. 
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e In addition to operating training programs, grantees implemented a variety of capacity-building 
strategies to accelerate and enhance learning, help participants persist in and complete their 
programs, and connect participants to employment: 


m= — Accelerated and enhanced learning strategies to reduce adult learners’ time to complete 
training programs. 


eRe _ = Persistence and completion strategies to support adult learners’ enrollment in, progress 
Sir toward, and completion of training programs. 


=) — Connections to employment strategies to connect adult learners to the workforce, such 
= as work-based learning strategies. 


The analyses presented in this report use data collected from participants at two points in time: program 
entry, generally the first day of class; and a follow-up survey conducted, on average, 15 months later. 
The analyses also use data collected from administrative wage records for a slightly longer period of time 
(five quarters after the quarter of enrollment). The research team selected the follow-up period for the 
survey to allow time for participants to complete programs (most of which, as noted above, were short- 
term) and enter employment, and also to provide sufficient time for analysis and reporting within the 
five-year evaluation period. 


Although the study reports on a wide variety of participant outcomes, to answer the final three research 
questions (whether some subgroups had better outcomes than other subgroups, whether some services 
have stronger associations with favorable outcomes, and whether some programs had higher levels of 
favorable outcomes), the report focuses on a narrower set of participant outcomes. This specific focus 
was necessary given the large number of subgroups, services, and programs studied. The four focal 
outcomes for this exploration of outcome variation are: 


e Program completion within about a year of program entry (completion defined as finishing 
required classes and being awarded an associated credential); 


e Training-related employment within about a year of program entry (training-related defined as 
having a job that relates closely to the participant's training program) for those not still enrolled in 
the program at follow-up; 


e Change in earnings between the third quarter before program entry and the fifth quarter after 
entry for those still not enrolled in the program at follow-up; and 


e Receipt of public assistance benefits (defined as means-tested benefits such as Supplemental 
Nutrition Assistance Program (SNAP) or Temporary Assistance for Needy Families (TANF) or benefits 
from other federally-funded assistance programs such as Trade Readjustment Allowances and Social 
Security Disability Insurance) about a year after program entry. 


For each outcome, the report describes findings for the full study sample for which it is relevant, as well 
as how it varies by participants’ characteristics at program entry, the supportive services they received 
during the program, and the program attended. Throughout, this report uses the variation across the 
34 programs to project likely success ranges for future cohorts of participants in these programs or 
similar ones funded by grant programs like TAACCCT. 
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SUMMARY OF FINDINGS 


Implementation of Capacity-Building Strategies 


Grantees implemented a range of strategies in each of the three capacity-building areas. 


Accelerated and enhanced learning strategies were among the most common ‘oe —x 
strategies implemented. For example, all grantees implemented career pathways a 
programs or stackable credentials. Most implemented at least one technology-enabled 


learning strategy, such as online or hybrid learning. More than half implemented prior 
learning assessments or provided credit for prior learning. 


Persistence and completion strategies often included academic supports, such as 
one-on-one advising and tutoring. Examples of nonacademic supports included advisors 
who helped participants identify and address challenges to program persistence and 
completion and provided referrals to needed services. 


Connections to employment strategies included work-based learning 
opportunities and employment-related services. All grantees provided opportunities 
for skills practice, either in a physical training facility or in a virtual training center. All used 
grant funds to purchase equipment, and in many instances, to renovate spaces for training. 
Fewer grantees offered in-program work experience; those that did, generally provided internships. Most 
grantees provided job search services either informally (through instructors) or through dedicated grant- 
funded positions, including career coaches, job placement specialists, and career developers. 


Characteristics of Participants at Program Entry 


Study participants were diverse in terms of demographics, education, employment, household income, 
and public assistance benefits receipt at program entry, as illustrated in Exhibit ES-2. 


e The most study participants were male, never married, and diverse in terms of race/ethnicity and 
age. 


e Most participants (53 percent) had at least some postsecondary education at program entry. 
e More than half (56 percent) were working at program entry. 


e The majority (57 percent) were low income, in families with less than $30,000 in annual income in 
the prior year (not shown). More than 40 percent were in poverty.* Few, however, received public 
assistance benefits. 


4 Poverty status is an indicator of whether the household was below the federal poverty level. The research team 
constructed the measure from exact family income and the number of people in household, using 2017 federal poverty 
guidelines (https://aspe.hhs.gov/2017-poverty-quidelines). For example, the poverty level for a family of three was 
$20,420. See Appendix Section A.3 for discussion of imputation of exact family income. 
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Exhibit ES-2. Participant Characteristics at Program Entry 


os) Sex 


85% | Male 


Family 
Structure 


56% | Not living with a spouse/ 
partner or children 


Poverty 
Status 


43% | Below the poverty level 


@ «. 


23% | 20 or younger 
17% | 21 to 24 

30% | 25 to 34 

29% |35 or older 


Source: Baseline Information Form 


Race and 
Ethnicity 


48% | White (Non-Hispanic) 

23% | Black (Non-Hispanic) 

22% | Hispanic, any race 
4% | Asian (Non-Hispanic) 


4% © Other, including multirace 
(Non-Hispanic) 


Educational 

Attainment 

Less than high school 

High school or GED 

Some college but no degree 


Technical or 
vocational degree 


Associate's degree 


Bachelor's degree or more 


ES Employment 
Status 
56% | Working at one 


or more jobs 


30% | Not currently working, 
but worked in the past 
12 months 


Has not worked in 
past 12 months 


Has never worked 


Note: Sample includes all study participants (V = 2,767). 


Training Outcomes 


Most Common 
Industries of Last Job 


CURRENTLY EMPLOYED 
PARTICIPANTS 


Manufacturing 
Retail Tade 


Accomodation and 
Food Services 


CURRENTLY UNEMPLOYED 
PARTICIPANTS 
Manufacturing 
Construction 


Transportation and 
Warehousing 


Receipt of Public 
Assistance Benefits 
Received SNAP 
Received UI 
Received TRA 
Receieved TANF 


Program completion. About 15 months after program entry, most participants (59 percent) had 
finished taking the required courses for their program, but not all of them received an associated 


credential. About half (51 percent) had completed their program; that is, they finished their courses and 
earned an associated credential (Exhibit ES-3). Seventeen (17) percent were still enrolled in their initial 
program. About one quarter (24 percent) left their program without finishing the required classes. Of 
those who finished their first program's classes, 17 percent had enrolled in an additional training within 
the follow-up period of approximately 15 months. 
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Exhibit ES-3. Training Progress Outcomes 


1. STATUS | Finished 


CREDENTIAL 


TRAINING 


2. PLANS 


Enrolled in 
Outcomes Study 


100% 


Classes 


59% 


Earned 
Credential 


ee 7) 


Started 
Additional 
Training 
17% 


Started 
Additional 
Training 
17% 


Did Not Start 
Additional 
Training 
43% 


Source: Short-term participant follow-up survey 
Note: Sample includes all survey respondents (/V = 2,211). 
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Employment Outcomes 


Employment Placement, including Training-related employment. More than three quarters of 
participants were employed, generally in full-time positions with benefits, approximately 15 months after 
program entry. Employment rates were similar for those who finished their classes (82 percent) as for 
those who did not finish (78 percent). 


A smaller proportion of participants, however, were employed in a job closely related to their training 
program (Exhibit ES-4), Among those who finished classes, 40 percent had jobs closely related to 
training, while 41 percent were employed in a job not related to training. For those who did not finish, 
14 percent were employed in a job closely related to training, while 64 percent were employed in a job 
not related to training. 


Change in earnings. Changes in earnings varied according to whether participants finished their 
classes. As one would expect, the change in earnings between three quarters before program entry and 
five quarters after was lowest for those still enrolled (Exhibit ES-5). Much more surprisingly, the change 
in earnings during this time period was higher for those who left their program without finishing than for 
those who finished their classes. 


Exhibit ES-4. Employment Outcomes Exhibit ES-5. Change in Quarterly Earnings 
100% $3,000 
$2,715 
80% $2,500 
$2,224 

60% $2,000 

40% $1,500 $1,422 
20% $1,000 
0% $500 

Finished classes Left without 

finishing A 


A, Not employed 


: = esti enrolled in required classes 
fe Employed in a job not related to training 


BB Finished classes 
SS Employed in a job related to training 


Zz Left without finishing 
Source: short-term follow-up survey Source: National Directory of New Hires (NDNH) 
Note: Sample includes survey respondents who Note: Sample includes participants who responded to the 
finished classes (V = 1,316) or left without finishing Short-term participant follow-up survey and who provided 
(N= 519). Components may not sum to 100 due to a valid SSN (V = 1,869). 


rounding. 
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Public Assistance Outcomes 


Receipt of public assistance benefits. As Exhibit ES-6 shows, the percentage of participants in 
poverty decreased by 8 percentage points, from 42 to 34 percent, between program entry and follow-up 
approximately 15 months later. This decline does not appear to have reduced public assistance benefits 
receipt, which stayed level over this period. 


Exhibit ES-6. Receipt of Public Assistance Benefits at Program Enrollment and 15 Months Later 


45% 


42% 
40% 
34% 
35% 
30% 
25% 
20% 
0, 
15% 1% 11% 
10% 9% 8% 
5% 2% 1% S 2% 2% 
0% me «~~: mS 
Poverty TANF SNAP Poverty and TRA 
receiving either 
TANF or SNAP 


my Program Entry NS Follow-up 


Source: short-term follow-up survey 
Note: Sample includes all survey respondents (=2,211). 


Participant Outcomes by Participant Characteristics and Services Received 


The research team explored how the four participant outcomes varied according to their characteristics at 
program entry and the services they received. The 10 characteristics, collected at program entry, were 
sex, age, race/ethnicity, Veteran status, TRA receipt, TANF and SNAP receipt, prior postsecondary 
education, time since last job, previous industry, and intended program duration. The 11 services 
received, collected by the participant short-term follow-up survey, were transfer credit and credit for prior 
learning (accelerated and enhanced learning strategies); career counseling, academic advising, financial 
advising, and workplace/life skills (persistence and completion strategies); and employment connections 
including skills practice in physical or virtual environment and in-program work experience (work-based 
learning strategies) and job placement services and interviewing practice (employment-related services). 
The team also explored how outcomes varied across participant groups defined by bundles of services. 


Certain participant characteristics and program services were associated with better outcomes, although 
not consistently across outcomes, as shown in Exhibit ES-7. 
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Exhibit ES-7. Participant Outcomes by Participant Characteristics and Services Received 


Program Training-related Change Receipt of Public 
Completion Employment in Earnings Assistance Benefits 
Participant Characteristics at Program Entry 
Sex 
Race/ethnicity (Black, non-Hispanic) YQ) 


Veteran Status 


TRA receipt 


SNAP receipt WY) 
TANF receipt 


Prior postsecondary education 


Age (25 to 34) 


Age (35 and older) 


Worked in past year 


Did not work in past year YQ) 


©© 
o 669 9800909 


Previous industry* 


S© ©0606 


Program is longer than one semester 


Transfer credits 0 


Credit for prior learning 


© 


Career counseling 


Academic advising a 


Financial aid advising 


Study, workplace, or life skills course 


Skills practice in work-like @® 
physical environments 


Skills practice in virtual environments 
Offered work study job or internship 


Job search or placement assistance 


SOO090 & 
© 


Interviewing practice services 


Source: Baseline Information Form and short-term follow-up survey 

Notes: Sample includes all survey respondents (W=2,211). An upward-pointing arrow indicates that the participant 
characteristic or service is associated with a higher outcome level. A downward-pointing arrow indicates that the 
participant characteristic or service is associated with a lower outcome level. 

*Previous employment in manufacturing is associated with a higher rate of training-related employment. Prior 
employment in professional and business services is associated with a smaller increase in earnings. 
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e Age was associated with higher rates of program completion and training-related employment for 
older participants (age 25 or above), but a stronger growth in earnings for younger participants. 
Older participants were more likely to receive public assistance benefits approximately 15 months 
after program entry. 


e Prior postsecondary education at program entry was associated with higher rates of program 
completion and training-related employment. 


e Employment status at program entry was associated with training-related employment, change in 
earnings, and public assistance benefits receipt. Specifically, participants with recent employment 
experience and those who previously worked in manufacturing had higher rates of training-related 
employment. Those who were not employed at program entry but had worked in the past year had 
weaker growth in earnings. Participants not employed in the year prior to program entry had higher 
rates of public assistance benefits receipt. 


e Public assistance benefits receipt at program entry was associated with lower rates of training- 
related employment, weaker growth in earnings, and higher receipt of public assistance benefits at 
follow-up. 


e Program duration was associated with stronger growth in earnings. The longest programs in the 
outcomes study—information technology programs—were among those associated with the largest 
earnings increases despite low completion rates. This pattern may be due to the fact that jobs in the 
information technology sector typically demand higher wages than those in some of the other sectors 
that were included in the study. 


e The combination of work-based learning and employment-related services was strongly 
associated with higher rates of program completion and training-related employment. 


e Skills practices in work-like physical environments was associated with higher rates of 
program completion and training-related employment. 


e Transfer credits, all three work-based learning services, and both employment related 
services was associated with higher rates of training-related employment. 


e Transfer credits had a positive association with earnings growth among participants not still 
enrolled in the program as of follow-up. 


e Different types of advising had effects on outcomes, Financial advising was associated with higher 
training-related employment rates and higher public assistance benefits rates. Academic advising was 
also associated higher public assistance benefits rates and lower program completion rates. 
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Participant Outcomes by Program 


Finally, the study explored participant outcomes by each of the 34 programs. Because most programs 
had small sample sizes, making it difficult to predict outcomes for future cohorts, the research team used 
new methods of statistical modelling to estimate outcomes and plausible ranges by program.°* 


Few programs ranked highly successful across all four outcomes with high levels of 
program completion, training-related employment, and change in earnings and a low 
level of public assistance receipt. In general, program rankings varied by the outcome in 
question. Only one program (welding technology at Washburn University) ranked in the top third 
across all four outcomes. Four other programs—mechanical craft (Chaffey College), licensed practical 
nurse (Washburn University), advanced manufacturing and Engineering (Chaffey College), and 
machining (South Central College)—ranked in the top third on three of the four outcomes. 


Seven programs that ranked low in program completion rate and training-related 
employment were nonetheless in the top third on change in earnings amongst 
participants who had left their programs. All seven of these programs were in the information 
technology sector. These were amongst the longest programs in the outcomes study, ranging in 
length between 14 and 24 months. 


Four programs that ranked high in program completion rate and training-related 
employment were in the middle or bottom third on change in earnings. These programs 
were mechanical craft, advanced manufacturing and engineering, and industrial automation at the 
Chaffey College consortium; and advanced manufacturing and welding technology at the Manchester 
Community College consortium. The programs ranged in length from 5 to 8 months. 


Discussion 


The analyses suggest a few areas for further exploration. 


Almost 60 percent of participants finished their classes, but only 51 percent of all participants 

(86 percent of finishers) completed their program by earning a credential. The combination of work- 
based learning and employment-related services had a positive effect on program completion rate. 
One particular service that program staff expected to help participants persist in and complete their 
program—academic advising—was associated with lower completion rates. A likely explanation for 
this result is “reverse causation”; that is, participants already struggling with their coursework may 
have been more likely to seek out academic advising than participants who were not struggling. 


Only 40 percent of participants who finished their classes or left their program without finishing the 
required classes were employed in a job closely related to their training at follow-up. As with program 
completion, the combination of work-based learning and employment-related services had a strong, 


5 


The methods used shift extreme outcome rates based on small sample sizes closer to overall outcome rates. This type of 
approach is common in the rating of doctors, schools, and teachers, but particularly so in the rating of hospitals. For 
some of the debate over hospital rating, see for example: https://www.cms.gov/Medicare/Quality-Initiatives-Patient- 


Assessment-Instruments/MMS/Downloads/Overall-Hospital-Quality-Star-Rating-on-Hospital-Compare-Public-Input- 


Period.pdf. It is less commonly used for local labor force market outcomes, but the European Union has been working 
on this and several member countries, such as Portugal and Italy, use these methods in their official local publications. 
https: //ec.europa.eu/eurostat/documents/3859598/10167610/KS-GQ-19-011-EN-N.pdf/3b56be5d-8266-0ee7-7579- 
7¢3e8e63289d. 
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positive effect on training-related employment. A potential area for future research is whether certain 
services in these areas contribute to positive outcomes more than others. 


e Change in earnings over a two year period from the 3" quarter before enrollment to the 5" quarter 
after enrollment was higher for participants who left their program before finishing their classes than 
for those who finished their classes. This somewhat surprising pattern appears to have arisen 
because those who stopped without finishing had lower pre-training earnings. Both groups had 
similar earnings at the fifth quarter after enrollment. It is not clear how people who stopped were 
able to catch up to the finishers despite starting from a lower level of earnings before training. 
Separately, it was shown that transfer credit receipt and work-based learning were associated with 
larger increases in earnings. It is possible that previous educational experience combined with new 
skills practice or in-program employment are enough of a foundation for participants to achieve 
higher earnings without actually finishing the training. More research would be needed to confirm this 
speculation. 


e Although poverty dropped over the study period, public assistance benefits receipt did not. One 
possible explanation is that participants were unaware at enrollment of the public assistance benefits 
to which they were entitled, and that the programs helped them sign up for benefits even as the 
poverty rate dropped. More research is needed to confirm this theory. 


This study explored participant and program outcomes at a point in time. Five quarters of follow-up is not 
likely sufficient to capture changes in earnings, which may take time to emerge, especially among 
participants enrolled in longer programs than the follow-up period (e.g., a number of IT programs, a 
welding program, and a mechatronics program). Relatedly, public assistance benefits receipt will likely 
decline as earnings grow, thus longer follow-up might provide more positive findings for that outcome. A 
longer follow-up period might also show higher rates of program completion. Year Up, a well-known 
training program found earnings impacts as early as Q5 (Fein and Hamadyk 2018), the same endpoint 
measured in this study, but Project QUEST did not detect an impact until year 4 after randomization,® so 
it would be interesting to follow-up on this sample at least through Q8 and possibly as far as Q16. 


At this point in time, services related to positive outcomes included work-based learning, employment- 
related services, transfer credits, and certain types of advising. Further study could help illuminate what 
about these services is particularly effective. 


6 Roder and Elliott (2019). https: //economicmobilitycorp.org/wp-content/uploads/2019/04/NineYearGains_web.pdf 
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1. Introduction 


The Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program 
awarded $1.9 billion to institutions of higher education that offer programs of two years or less, mostly 
community colleges. The aim was to build their capacity to provide workforce education and training to 
adults in need of new skills for in-demand jobs, including those eligible for training under the Trade 
Adjustment Assistance (TAA) program as well as unemployed workers and other adult learners. The grant 
program, which ran from 2011 to 2018, was also designed to address other key issues—changing 
education and workforce systems to be better connected and integrated, more effectively addressing 
employer needs for skilled workers, and transforming how community colleges deliver education and 
training to adult learners.” 


The fourth and final round of grants was awarded in 2014 and ended in 2018. This report describes 
education, earnings, and self-sufficiency outcomes of participants about 15 months after entry into 

34 programs at nine selected Round 4 grantees. It also describes the implementation of those grantees’ 
programs. This outcomes study contributes to the body of evidence accumulated from the TAACCCT 
national evaluation across all four rounds of grantees. 


1.1. OVERVIEW OF THE TAACCCT GRANT PROGRAM AND 


NATIONAL EVALUATION Grantees and Colleges by 
Across all four rounds, TAACCCT funded 256 grantees Grant Round 
comprising colleges in all 50 states, the District of 
Columbia, and Puerto Rico.® Grantees could be single Round 1: 49 grantees, 353 colleges 
institutions or lead institutions of a consortium of colleges.® Round 2: 79 grantees, 310 colleges 


The TAACCCT grant program aimed to: Round 3: 57 grantees, 187 colleges 


Round 4: 71 grantees, 263 colleges 


e Better prepare TAA-eligible workers and other adults 
for high-wage, high-skill employment or reemployment 
in growth industry sectors by increasing their 
attainment of degrees, certificates, diplomas, and 
other industry-recognized credentials that match the skills needed by employers; 


Approximately 729 unduplicated 
colleges across all rounds 


7 Congress authorized the TAACCCT grant program as part of the American Recovery and Reinvestment Act of 2009 to 
increase the capacity of community colleges to meet local and regional labor demand for a skilled workforce. The Health 
Care and Education Reconciliation Act of 2010 provided the TAACCCT program with $1.9 billion in funding over fiscal 
years 2011-2014, approximately $500 million annually over four rounds of grants. 


8 — Eligible institutions were institutions of higher education, as defined in the Higher Education Act of 1965 as amended 
(20 USC 1002), that offer programs that can be completed in two years or less. Institutions of higher education include 
public, proprietary, or nonprofit educational institutions or postsecondary vocational institutions. 


9 Funding varied by the type of grantee. In Round 4, grant awards for single-institution grantees ranged between $2.32 
and $3.25 million, whereas consortium grantees received between $6.44 and $20.0 million. Three single grantees and 
four consortia were awarded funding at a level that exceeded DOL funding cap guidelines. Such activities could include 
those that “(1) Advance State Career Pathway Systems; (2) Improve Statewide Data Collection, Integration, and Use; or 
(3) Create Nationally Recognized Competencies and Credentials” (https://www.doleta.gov/grants/pdf/SGA-DFA-PY-13- 
10.pdf, pp. 20-27). 
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e Introduce or replicate innovative and effective methods for designing and delivering instruction that 
addresses specific industry needs and leads to improved learning, completion, and other outcomes 
for TAA-eligible workers and other adults; and 


e Demonstrate improved employment outcomes for grant-supported program participants. 


Grantees used three types of capacity-building strategies to connect program participants to 
employment: strategies that supported accelerated learning and enhanced learning to reduce adult 
learners’ time to completing a program of study;'° persistence and completion strategies to support 
adult learners’ enrollment, progress, and completion of programs; and employment connection 
strategies, including work-based learning and employment services. 


In order to build evidence on its grant-funded programs and strategies, DOL funded a national evaluation 
of each round of grants to collect and assess data across all participating colleges. The Round 4 national 
evaluation included an outcomes study, the focus of this report. 1! 


TAACCCT National 
Evaluation Components 


is (Rounds 1-4) of the service delivery approaches developed 
and the systems changed through the grants. 


An /y of nine Round 4 grantees and 34 programs using survey data and 
administrative records to better understand the characteristics of participants, their service 
receipt, and their training and employment outcomes. 


5 (Rounds 1—4) to develop a national 


Scie of the mpemeneten of the capacity-building strategies and build evidence of the 
effectiveness of the strategies on participants’ training and employment outcomes. 


> with selected Round 4 employer partners, to better 
undetand isan hae perspectives on how to develop and maintain strong relationships 
with colleges. 


1.2. OUTCOMES STUDY OVERVIEW 


The outcomes study describes the characteristics of 2,767 participants who enrolled in 34 programs 
across nine selected grantees. Specifically, it documents participants’ characteristics; service receipt; and 
outcomes related to training progress, employment, and receipt of public assistance benefits at roughly 
15 months after participants’ enrollment into their programs. !* It also documents earnings growth from 
the third quarter before enrollment through the fifth calendar quarter following the quarter of enrollment. 


10 For purposes of this report, “completing” a program means finishing the required classes and earning a targeted 
credential. 

4 Findings from the implementation study are included in the College Survey Report and this report; the implementation 
study is not a stand-alone report. 

22 The survey was fielded 12 months after program entry, and many interviews were administered starting at month 12, 
but interviewing efforts continued for many additional months in order to boost response rates. Median lag from 
program entry to interview was 14.8 months, with 90 percent of interviews occurring between 12 and 18 months after 
program entry. 
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The research team selected the 34 programs for the outcomes study using the following criteria: a 
program had to (1) be funded or expanded with the grant; (2) be short-term to allow for follow-up 
starting at 12 months post program entry; (3) result in a credential; and (4) have a distinct point of 


entry. 


Exhibit 1-1 below shows the location of the nine selected grantees. 


Exhibit 1-1. Round 4 Grantees in the Outcomes Study 


CA 


Chaffey College 


J Single Institution 
Grantees 


MN 
Manchester 
, Community College —@ 
South Central cr 
College 
OH 
IN 
KS Cincinnati State 
Ivy Tech Technical and 
Washburn Community Community College 
: = College 
University 
Bossier Parish _| 
Community College an 
FL 
Delgado 
Community 
College 
Miami Dade 
College 


The analyses presented in this report use data collected from study participants at two points in time 
(program entry and approximately 15 months later), as well as administrative wage records (see 

Chapter 2 for more detail). The report not only summarizes the outcomes of program participants, it 
provides information of use for policymakers and program operators interested in funding, designing, and 
implementing college-based programs to train dislocated and other workers. 


Analyses in this report: 


e Document the capacity-building strategies implemented by grantees; 


e Describe the characteristics of adults who enrolled in the grantees’ programs; 


13 


While the research team intended to focus on short-term training programs, it had to make some exceptions to increase 


the grantee and program-level sample sizes. In some cases, such as Ivy Tech's information technology program, study 
intake was conducted in introductory level courses that could culminate in a credential, but most often laddered into 
medium to longer-term training programs. The team weighed the tradeoff between program length and sufficient 
sample to conduct analyses when selecting programs for inclusion in the study. 
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e Document the program completion, employment, earnings, and public assistance benefits receipt 
outcomes of participants a year after program entry; 


e Explore how these outcomes vary by participant characteristics at program entry; 


e Explore which grant-funded services are more or less helpful to program participants in achieving 
favorable program completion, employment, earnings, and public assistance benefits receipt 
outcomes; and 


e Explore the variation in participant outcomes by program. 


Readers of this report may be interested in extrapolating the information reported here to new settings. 
To assist readers, this report contains not just results, but measures of how consistent the results are 
across participants and programs. Consider the example of future earnings. If 80 percent of participants 
experienced two-year earnings growth between $200 and $800 at one program but the comparable 
range is $2,300 to $3,900 at another program, then extrapolation to a new program will involve much 
more uncertainty than if 80 percent of students at every program experienced earnings growth of $2,000 
to $2,400. 


To address this, the study uses the variability in outcomes across participants and programs to speculate 
about how well the experiences in these 34 programs would extrapolate to a hypothetical future round of 
grants. Instead of reporting that some percentage of participants obtained a job aligned with their 
training, the report provides ranges of how far average experiences from a new round of similar grants 
might plausibly vary from the average experiences reported here. 


Overview of Programs 
in the Outcomes Study 


e 34 programs offered by 9 grantees and located in 8 states (California, Connecticut, Florida, 
Indiana, Louisiana, Kansas, Minnesota, and Ohio) 


e 6 consortium grantees, 3 single-institution grantees 


e Programs spanning 6 industries: manufacturing (14 programs), information technology 
(9 programs), welding (5 programs), construction (3 programs), health care (2 programs), 
and transportation/warehousing (1 program) 


e 2,767 participants, ranging from 11 to 334 participants per program: average of 55 per 
program, median of 81 


e Program durations ranging from 1 week to 24 months: average of 8.7 months, median 
(0) i ofe) 
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1.3. GUIDE TO THE REPORT 


The remainder of the report includes the following chapters: 


Chapter 2 describes the outcomes study design and data sources. 


Chapter 3 documents the nine grantees’ capacity-building strategies implemented in the areas of 
accelerated and enhanced learning, persistence and completion, and connections to employment 
(work-based learning and employment services). In doing so, it provides useful context for 
understanding the outcomes of participants in the 34 selected programs. 


Chapter 4 begins the outcomes analyses. It first describes the characteristics of study participants 
and then discusses training duration and service receipt. The chapter then reports participants’ 
outcomes related to training progress, employment, earnings, and public assistance benefits receipt 
approximately 15 months year after program entry. 


Chapter 5 explores how outcomes vary across subgroups of participants defined by their 
demographic, economic, and education characteristics at program entry. 


Chapter 6 explores connections between outcomes and grant-funded services participants received 
after program entry. The methodology for these analyses focuses on the effect of use of a service by 
participants, rather than the offer of a service by the college. Thus, it compares outcomes for 
participants who reported use of the service versus those who reported not using the service. 


Chapter 7 summarizes outcomes by program. It ranks the 34 programs relative to one another on 
program completion, employment, earnings, and public assistance benefits receipt outcomes. 


Chapter 8 reflects on the findings across chapters. 


The report includes the following appendices: 


Appendix A documents data collection about the characteristics and experiences of program 
participants when they enrolled in the study and associated statistical analyses. Appendix B 
documents data collection about the program and post-program experiences of program participants 
and associated statistical analyses. Appendix C documents data collection about earnings of program 
participants and associated statistical analyses. Appendix D gives technical details on how the team 
estimated the adjusted differences in outcomes by receipt of grant-funded services. Appendix E 
gives technical details on how the team estimated the program outcomes separately for each 
program. Appendix F documents how data about the programs were collected directly from grantee 
and program staff and associated statistical analyses. Appendix G provides supplemental detailed 
tables. 
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2. Study Design and Data Sources 


This chapter describes the research questions, data sources, and analysis methods for the outcomes 
study. ** It begins with the TAACCCT capacity-building theory of change, which underlies the research 
questions and data collection efforts. 


The implementation analysis is designed to document the implementation of grant-funded strategies. The 
outcomes analysis is designed to describe the characteristics of participants in the 34 programs who 
agreed to be part of the study; their training and service receipt; and various education, employment, 
earnings, and public assistance benefits receipt outcomes about 15 months after program entry. It then 
explores program performance by participant outcomes. Finally, the chapter discusses the study’s 
limitations. 


2.1. CAPACITY-BUILDING THEORY OF CHANGE 


Exhibit 2-1 depicts the TAACCCT capacity-building theory of change.!* Such theories of change provide 
an organizing framework for assessing the implementation of grant activities and how they are 
hypothesized to lead to improved outcomes for participants and grantees. Exhibit 2-1 highlights 
outcomes experienced by participants. Although grantees also had capacity-building outcomes, 
participant outcomes are highlighted here because they are the focus of this report. 16 


Starting at the left, the exhibit shows three types of program inputs. First are the grantees’ efforts to 
develop and enhance their education and training programs and build capacity to serve adult learners. 
Grantees built on existing programs and physical infrastructure, such as classrooms, labs, and skills 
practice environments where training occurs, and on institutional infrastructure, such as faculty. 
Contributing to those effects are partnerships, which could be internal to the college, such as campus 
services, administrators, and other departments, and external to the college, such as employers and the 
public workforce system. 


The second type of input is the participants who need skill upgrades or new training to compete in the 
labor market or advance in their careers. Participants include TAA-eligible workers and other adult 
learners who need training, such as veterans and incumbent workers. 


14 Designs for the implementation study and the outcomes study are described in detail in the Eva/uation Design Report 
(Abt Associates 2017). 


15 This theory of change builds from the frameworks presented in the Rounds 1 and 2 and Round 3 TAACCCT synthesis 
reports (Eyster 2020; Kuehn and Eyster 2020; Scott et al. 2020). It also builds on theories of change developed for 
other recent evaluations of occupational training programs (see, for example, Fein 2012; Werner et al 2013). 


16 For detail on the outcomes experienced by the colleges, see (Trutko et al. 2020). 
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Exhibit 2-1. Theory of Change 


PROGRAM INPUTS INTERMEDIATE OUTCOMES LONGER-TERM OUTCOMES 


Grantee/College Vas test ol-balec] sXe hber-teCosat-V Employment Educational Employment 


ttai t ttai t 
¢ TAA-eligible workers oie cine e.g., employed in a job 


Existing programs e.g., employed in a job 


« Internal partners * Veterans e.g, completed aligned with training eg, completed aligned with training 
* TAACCCT funding © Other adult learners program and received additional program(s) 
* Other partners targeted by target credential and received credentials 


* Employers 

« WIBs 

« External service 
providers 


grantees/colleges 


Bs Co}bt-t—V alo) Co Bbatoroy eat) bs Copbl-r Valo} Co balers an) 
and receipt of and receipt of 
polbte)bteMol—sat—vani] public benefits 


CAPACITY-BUILDING STRATEGIES 


e.g., increased earnings e.g., increased earnings 


Accelerated Persistence and between the 3rd between the 3rd 
and Enhanced Completion-Related quarter before program ; quarter before program ‘ 

3 P enrollment to the 5th e.g., decreased receipt enrollment to the 5th €.g., decreased receipt 
Learning Related quarter after enrollment of public benefits quarter after enrollment of public benefits 


* Redesign of * Academic and 
curriculum, nonacademic support 
credentials, and services 
programs « Redesign of 

Alignment of college developmental and 
Policies adult education 

* Technology-enabled Transfer and 
learning articulation 


Persistence and 
Completion-Related 


© Work-based learning 

* Employment-related 
services 

Partnerships with 

employers and industry 

and the public 

workforce system 


CONTEXTUAL FACTORS: Local economy; size of target population; college budgets; other training providers in area; state and local policy and funding 
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The third type of input is capacity-building strategies: 


Q — Accelerated and enhanced learning strategies to reduce adult learners’ time 
= to complete training programs, such as curriculum redesign to create career 
AS pathways or stackable credentials, technology-enabled learning, transfer credits, 

and credits for prior learning. 


S e Persistence and completion strategies to support adult learners’ enrollment in, 
e © progress in, and completion of training programs, such as academic advising 

(e.g., tutoring), nonacademic advising (case management), financial aid advising, 
career counseling, and nonacademic skills courses (e.g., life skills). 


e Connections to employment strategies to connect adult learners to the 
workforce, such as work-based learning strategies (skills practice in physical or 
virtual environments, internships and apprenticeships) and employment services 
(job search or placement assistance and interviewing practice). 


The middle column in the framework is intermediate outcomes for participants; that is how grant- 
funded activities were expected to affect participants in the short-term. This includes program 
completion; employment closely related to training; increased earnings; and reduced use of public 
assistance benefits. 


The column on the right shows longer-term outcomes for participants who complete their programs. 
These include enrolling in additional trainings, obtaining additional credentials, progressing in their 
employment, and further increasing their earnings. 


Finally, contextual factors can influence both the implementation of grant activities and its outcomes. 
For example, local economic conditions can affect the supply of potential participants for programs as 
well as the availability of jobs for program completers. 


2.2. DESIGN FOR IMPLEMENTATION ANALYSIS 


The implementation analysis (Chapter 3) assesses the design and implementation of the grant-funded 
activities. It focuses on program inputs, specifically the nine grantees and their implementation of the 
capacity-building strategies. The theory of change posits that inputs should affect intermediate outcomes, 
which are the focus of the outcomes study. Specifically, the implementation analysis explores the 
following research question: 


What were the primary grant-funded components of each of the nine outcomes study 
grantees and how were they implemented? 
The implementation analysis explores the design of the 34 programs selected to be included in the 
outcomes study, including the nature of their training delivery, supportive services available, and 
employment supports for program participants and completers. 


The implementation analysis uses data collected using interviews and observations conducted during site 
visits to each of the nine grantees. Site visits occurred during Year 3 of the grants (March and April 
2017), when programs and other grant activities were expected to be operating at a steady state. During 
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the visits, the research team met with staff associated with all 34 programs, which, depending on the 
grantee, was all its grant-funded programs or a subset of them.” 


2.3. DESIGN FOR OUTCOMES ANALYSIS 


The outcomes analysis documents the characteristics, experiences, education, earnings, income, and 
public assistance benefit receipt outcomes for program participants who enrolled in the 34 programs and 
agreed to participate in the study (see Enrolling Study Participants box). The outcomes study 
assesses program inputs, specifically the economic, demographic, and educational characteristics of the 
program participants, and intermediate outcomes. 


Chapter 4 reports participant results for a number of intermediate outcomes, whereas subsequent 
chapters focus on four specific outcomes for detailed analyses by participant characteristic at program 
entry (Chapter 5), services received (Chapter 6), and program attended (Chapter 7).18 The four focal 
outcomes are: 


e Program completion: finishing the required classes and earning an associated credential within a 
year of program entry;'? 


e Training-related employment. employment in positions closely related to training (among 
participants who are not still enrolled in their program at the time of follow-up); 


e Change in earnings: change in earnings between the third quarter prior to program entry and the 
fifth quarter following program entry2° (among participants who were fof still enrolled in their 
program at the time of follow-up); and 


e Public assistance benefit receipt. receipt of means-tested public assistance (including Temporary 
Assistance for Needy Families and Supplemental Nutrition Assistance Program) as well as other 
federally funded assistance programs (e.g., TRA and Social Security Disability Insurance). 


The team selected these four because they are the central intermediate outcomes in the theory of 
change, and because they range from direct and early (e.g., program completion) to indirect and distal 
(e.g., public assistance benefit receipt). The underlying logic is that participants who complete their 
program and obtain training-related employment should experience earnings growth such that they 
would not need public assistance benefits. 


17 Some programs did not lend themselves to inclusion in the outcomes study, most often because there was no 
identifiable point of study intake. The team conducted calls with grant staff in spring 2016 to identify grantees and 
programs to include in the outcomes study. Selected programs had to meet the criteria discussed in Section 1.2 above. 


18 The team narrowed to focus on four outcomes for presentation purposes. There are so many ways that each of the four 
outcomes is analyzed that the volume of analyses and their presentation would become unmanageable for a larger set 
of outcomes. 


19 Participants’ program entry date was usually the first day of class. 
20 This period between Q(—3) and Q5 was the longest period available for the entire sample of study participants. 
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Enrolling Study 
Participants 


Colleges were responsible for enrolling training program participants in the outcomes study. 
The research team trained college staff on how to describe the study, explain informed 
consent, and administer a Baseline Information Form that collected participants’ 
(a{=unlole] ¢-] 9) ql (omm=rere)ale)snl(onmrs) ave =rel0 (etx) u(0)at=) i oy=(ol,<e] cella (eM (a) co)auate]u(e)pmuo mag lem ale M@e)a =~ 0) K-16 mine) 
be in the study. 


Staff administered the informed consent form and the Baseline Information Form in class at 
the beginning of the training program. All study participants attended at least one class 
session of their program. Staff described the study to all program participants enrolled in the 
target programs. 


Program entry (enrollment in the study) began in August 2016 and ended in October 2017. 
Grantees’ reported consent rates varied by program and ranged from a high of 100 percent 
to a low of 60 percent. 


The outcomes study explores these specific research questions: 


1. 


What were the characteristics of study participants? 


2. In which features of training programs and services did study participants engage? What 


college and partner (e.g., workforce agency) services did they receive? What were their 
assessments of the services they received? 


What education outcomes did study participants in short-term training programs 
achieve? Did they see themselves on a career pathway? What were their plans for future 
training? What was the expected timeline? 


Did participants obtain employment? If so, was this employment in the occupation for 
which they trained? What were their earnings? How did it compare to their earnings 
Prior to entering their training programs? What were the characteristics of their jobs? 


Did household income change? Were study participants less likely to receive public 
assistance benefits at follow-up than at program entry? 


Which subgroups of participants were most likely to complete grant-funded programs? 
Begin jobs in their occupation of training? Have higher post-training earnings? 

Do study participants who utilize various services have better outcomes, after controlling 
for characteristics at program entry? 


How did success rates (program completion, training-related employment, change in 
earnings, and public assistance benefits receipt) vary across programs and grantees? 
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2.4. DATA SOURCES 


Three data sources supported the outcomes study:2! 


e Baseline Information Form. Study participants who consented to be in the study completed a 
Baseline Information Form, which collected detailed demographic and socioeconomic characteristics, 
including educational and training background. In addition, it collected each study participant's 
employment history as well as information on current wages and earnings, public assistance benefits 
receipt, and total household income. The team modeled the Baseline Information Form on forms 
developed for evaluations of other DOL grant programs.*? Other than counts by grantee (see 
Appendix Section A.1), no data were retained about the participants who did not consent to be in the 
study. 


e Participant Follow-Up Survey. The follow-up survey, administered approximately 15 months 
following program entry, collected information on training receipt, perceived training quality, and 
educational progress; training-related supports; employment characteristics; and income and public 
assistance benefit receipt. Survey administration started 12 months after program entry, and many 
interviews were administered in month 12. However, interviewing efforts continued for additional 
months to increase response rates. ?3 


e National Directory of New Hires (NDNH). This national database includes new hire dates, 
quarterly wages, and Unemployment Insurance benefit data.24 The team used NDNH quarterly wage 
data to assess study participants’ earnings and employment status. Because the NDNH captures 
information for all jobs covered by Unemployment Insurance as well as most federal jobs, for the 
outcomes study it included near-complete coverage of all formal employment for study participants 
who provided a valid Social Security number for matching.*° NDNH data analysis focused on quarterly 
employment levels and earnings three quarters prior to program entry and five quarters after 
program entry, which was the longest period available for the entire sample. 


2.5. ANALYSIS METHODS 


The research questions required varied analysis methods. As discussed in Section 1.2, the research team 
estimated average outcomes for participants, as well as ranges for estimates that reflect plausible errors 
in extrapolating from these results to a hypothetical future round of TAACCCT-like grants. To answer 
Research Question 7 about the relationships between services and outcomes, the team speculated about 
what these relationships would be if participants opting to use a given service had the same baseline 
profile as those opting not to use it. Finally, to answer Research Question 8 about the variability in 


2 


are 


Data collection instruments are available at https://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=201505-1291-001 

and https://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=201608-1291-001. 

22 Those evaluations are Evaluation of Green Jobs and Health Care Grantees and the Evaluation of the Impact of the H-1B 
Technical Skills Training Grants. 

23 The team used three survey modes: web administration, followed by telephone follow-up, and finally in-person follow- 
up. The survey had an 80 percent response rate. Most of the interviews took place between 12 and 15 months after the 
participant’s program entry date (which was usually the first day of class). The median lag was 14.8 months. However, 
6 percent of interviews took place as soon as 10 or 11 months after program entry, and 33 percent took place 
16 months or longer after program entry. The longest lag from program entry to interview was just less than two years. 

24 The NDNH is maintained by the Office of Child Support Enforcement in the U.S. Department of Health and Human 

Services. 


25 About 85 percent of participants provided a valid SSN for matching with the NDNH. 
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outcomes across programs, the team speculated about how these differences would have diminished if 
the sample sizes had been larger in each program. More details are provided below and technical details 
are included in the appendices. 


To answer Research Questions 1 through 5 (Chapter 4), the research team calculated simple 
averages and percentages along with corresponding estimated standard errors and 95 percent 
confidence intervals for all participants. Given the small sample sizes for many of the programs, the 
team ran the tabulations on the pooled set of 34 programs. Standard errors for the pooled estimates 
reflect variation in success rates across the programs as well as variation within programs. Based on 
these standard errors, the team computed 95 percent confidence intervals for each outcome. A 
confidence interval is a measure of uncertainty. Though there is very little uncertainty about 
outcomes for the participants who enrolled in the 34 programs during the study period, there is 
considerable uncertainty about outcomes for future participants—especially what outcomes might be 
for those who enroll in programs offered by new grantees if there were another round of grants or 
similar type of grant programs. The confidence intervals in this report represent the range of possible 
outcome levels that would be expected for programs from a new round of grants, assuming the same 
general economic conditions and same grant competition rules.*° For more information on how to 
interpret these confidence intervals and on how they were calculated, please see Appendix 

Sections A.4 and B.5.27 


To answer Research Question 6 (Chapter 5), the team estimated outcomes for a set of subgroups 
defined by participants’ demographic, economic, and educational characteristics at program entry. 
Differences between complementary subgroups (e.g., males and females) were formally tested to 
determine whether differences are large enough relative to sampling error to form a reasonable basis 
for extrapolation to future hypothetical grants, assuming the same general economic conditions and 
same grant competition rules. The objective of subgroup analyses is to determine which groups 
consistently benefit more from these sorts of training programs.?° 


To answer Research Question 7 (Chapter 6), the team explored which grant-funded services were 
associated with better and worse outcomes for participants, after accounting for known differences at 
study enrollment. The answers to this research question are more tentative than those to the earlier 
research questions. In order to determine whether receiving a service during training leads to a 
better outcome than not receiving it, the research team used different analytical methods than for 
the prior research questions. These methods make strong implicit assumptions about what led 
participants to receive the services. More specifically, based on characteristics of participants at 
program entry and on the nature of the program, the research team built statistical models for 
“counterfactual” outcomes—the outcomes that participants who received the service would have 
experienced if they had not received the service—as well as for the outcomes that participants who 
did not receive the service would have experienced if they Aad received the service. If these models 
are accurate, then the estimated impacts should be useful as a guide to future program design. 


26 


27 


28 


These intervals would not apply during the COVID-19 pandemic, but could do so in a few years, near the end of the 
next expansionary business cycle. 


These confidence intervals are narrower for outcomes with similar averages across programs and wider for outcomes 
with highly disparate averages across programs. The logic is that if the 34 programs all led to similar averages for an 
outcome, then a future round of grantees should also lead to a narrow range of averages for that outcome; but if the 34 
programs lead to wildly different averages for an outcome, then it is much more difficult to predict the average for that 
outcome for a future round of grants. 


If the difference between two subgroups is consistent across the 34 programs, then the difference between the 
subgroups can be statistically significant even if the confidence intervals for the two subgroups overlap. 
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Conversely, if these models are flawed, the estimated impacts may serve as a misleading guide to 
future program design. This methodology is equivalent to a series of “quasi-experimental designs” 
(QEDs) in which participants that received a service are the “treatment” group and those who did not 
are the “control” group, and regression analysis is used to remove observed confounding. Full details 
on these methods are provided in Appendix D. 


e Toanswer Research Question 8 (Chapter 7), the team used another type of modeling to estimate 
program-specific outcomes. The participant sample sizes were too small in most programs for the 
simple program-specific average to serve as a useful guide for the likely experiences of future cohorts 
of program participants. Instead, the team built models for what would have happened in each 
program if it had an adequate sample size; that is, if it enrolled a large number of participants similar 
to those who did enroll. Predictions from these models tend to shift extreme outcome rates based on 
small sample sizes closer to overall outcome rates. This methodology is equivalent “small area 
estimation” (SAE). Federal statistical agencies in a variety of countries use these methods to publish 
estimates for localities with small or even zero sample sizes. In this application, the programs serve 
as “areas.” Full details on these methods, including academic references to similar work, are provided 
in Appendix E. 


2.6. STUDY LIMITATIONS 


This outcomes study has some limitations. First, the analyses are based on the experiences of 
participants who enrolled in 34 programs at nine grantees. The research team selected these grantees 
and programs based on specific criteria, as noted above. Thus, the grantees and programs are not 
representative of all Round 4 grantees and grant-supported programs. This limits the generalizability of 
findings. Findings are generalizable to the extent that other grant-funded programs are similar to the 
programs selected for the study in terms of duration, credential timing, in possessing a specific point of 
entry, and in operating in a specific economic environment. The last point is particularly important given 
that the outcomes were mostly measured in 2018, a time period when unemployment was low. 


Second, the follow-up period is relatively short and many of the interviews were conducted during the 
preceding three months. Thus, caution should be used in interpreting the findings. Fifteen months after 
program entry is likely to be too early to capture important changes in employment and earnings 
outcomes. In fact, a few of the programs in the study were intended to last more than 12 months. Where 
appropriate, participants who were still in training were excluded from analyses. A longer reference 
period would allow the inclusion of such participants in those analyses. 


Third, except for the analysis in Chapter 6, the study lacks a control group, thus outcomes reported for 
the full study sample cannot be characterized as better or worse than participants would have achieved 
without access to the program.”? Other potential study limitations related to specific analyses are 
discussed in the relevant chapters. 


29 The research team explored the feasibility of conducting an experimental study with a control group, which was the 
primary reason that a key selection criterion was the program having a specific point of entry. The team abandoned this 
design once it was clear that demand for most programs was considerably lower than anticipated, thus making an 
experimental design infeasible. The team also considered developing a comparison group using data from participants in 
another study (or studies), but this approach ran into difficulties obtaining permission to use participant records from 
studies of sufficiently similar populations. 
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3. Implementation 


IMPLEMENTATION FINDINGS 


Findings 


This chapter describes grant-funded strategies 
implemented by the nine grantees in the 
outcomes study. It addresses the following 
research question: 7° 


What were the primary grant-funded 
components of the nine outcomes 
study grantees and how were they 
implemented? 
The chapter uses the three capacity-building 
strategy categories described in the theory of 
change as a framework for describing grantee 
implementation activities: acce/erated and 
enhanced learning, persistence and 
completion, and connections to 
employment (work-based learning and job 
search strategies). The substrategies within 
each capacity-building strategy category are 
informed by DOL priorities articulated in the 
Round 4 grant funding announcement. 


Though the focus of this report is on 
outcomes associated with the 34 programs, 
this chapter describes both grantee-level and 
program-level strategy implementation. 
Because grant-funded strategies often support 


Key Takeaways: 
Implementation Findings 


All grantees designed new or enhanced 
existing career pathways or developed new 
stackable credentials. 


Eight grantees implemented at least one 
technology-enabled learning strategy. 


Five grantees implemented prior learning 
assessments or awarded credit for prior 
ere] gallate 


All grantees implemented a new learning 
structure for their training programs. 


Grantees used grant funds to provide a range 
(0) mr-\er-6|=11 4) (Om=10]9) 910) 0-5 


Grantees implemented a range of 
nonacademic supports. 


All nine grantees implemented skills practice 
strategies. 


Seven grantees provided job search services. 


Internships were the most common work- 
based learning activities among grantees. 


multiple programs, overall grantee strategies are described first, followed by program-specific examples. 


This chapter begins with an overview of the 34 programs included in the outcomes study, including key 
strategies. It then describes grantees’ implementation of each capacity-building strategy. 


3.1. OVERVIEW OF PROGRAMS 


The nine grantees’ 34 programs spanned six industries: manufacturing, information technology (IT), 
welding, construction, health care, and transportation/warehousing. Exhibit 3-1 groups programs by 
industry and for each program summarizes its intended length, associated credential(s), and key 
strategies. The most common strategies implemented were related to the strategies of accelerated and 
enhanced learning and connections to employment. 


30 This chapter complements findings in a separate implementation report that used a survey administered to all Round 4 
colleges (Trutko et al. 2020). 
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Exhibit 3-1. Program Length, Credentials, and Strategies Used, by Industry 


eats Advanced Manufacturing 
ad 


Program and Grantee 


Program 
Length 
(Gite dit)) 


Credential Types 


IMPLEMENTATION FINDINGS 


Primary Strategies Implemented 


Fast Track to Manufacturing; 


MSSC Certified Production Technician 


Stacked & latticed credentials, hybrid & self- 


Bossier Parish Community : certification paced learning, skills training 
College 
Advanced Manufacturing and National Career Readiness Certificate stackable F . ‘ 
Engineering, Chaffey College d credential leading to a portfolio of credentials Skills practice, career boot camp, navigator 
EPA 608 license in refrigerant management Equipment for skills practice, internships, 
HVAC, Chaffey College 11 North American Technician Excellence accelerated & contextualized curriculum 
certification development 
Industrial Automation, Chaffey . : ” Accelerated and cohort-styled curriculum, ski//s 
College u Ind aie autora GH en erates: practice, employability skills workshop 
; : InTech Center (skills practice); 3 training 
rarniebey Maintenance Chaltey 3 NCCER Industrial Maintenance certificate pathways, stackable credentials, supportive 
Ceres services, job search assistance 
: : 2 Equipment for skills practice, updated 
Mechanical Craft, Chaffey College 5 Portfolio of NCCER certificates modularzed clilculum 
ae . ; re InTech Center (skills practice), supportive 
Pre-engineering, Chaffey College 3 Portfolio of NCCER certificates Wi ‘ob search acclctance 
Tey) Sot P Elgato Community 4 NCCER Core certificate 
College = ; ; 
mi r rticulation agreemen 
Eola: Delgado Community 4 NCCER Core certificate equipment, skills training, hybrid instruction 
seiaheed Mandailing: Advanced Manufacturing Machine Technology Facility renovation and equipment for skills 
Ne 8 certificate practice, stackable credentials, expanded 


Manchester Community College 


Precision Manufacturing certificate 


program to new colleges/target populations 
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Certified Production Technician, 
South Central College 


MSSC Certified Production Technician 
certification 


Machining, South Central College 


Examples include CNC Certificate 

Machine Operator Right Skills Now Certificate 
CNC Toolmaking Diploma 

Machining Technician Diploma 

CNC Precision Manufacturing Technician Diploma 
CNC Manufacturing Technician AAS 


Mechatronics, South Central 
College 


16 


Industrial Maintenance Certificate 
Mechatronics Diploma 

Automation and Robotics Systems Diploma 
Mechatronics AAS 

Automation and Robotics AAS 


Right Skills Now; South Central 
College 


Construction 


Program and Grantee 


Program 
Length 


Machine Operator Right Skills Now Certificate 


Credential Types 


Career pathways, online course, equipment for 
Skills practice in physical environment and 


virtual environment, academic advising, job 


search assistance 


Primary Strategies Implemented 


Training for Manufactured 


(Cited dit)) 


Dade College 


Construction (TRAMCON) Advanced, 6 NCCER Manufactured Construction Level 2 
Miami Dade College 

TRAMCON Basic, Miami Dade Se 

College 3 NCCER Level 1 certification 

Tee et SaURdanen eileen 3 NCCER Core certification 


Stackable credentials, 7ransformative Learning 
Centers (skills practice), 


job search assistance 
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Ce Health care 


Program 
Length 
(months) 


Program and Grantee Credential Types Primary Strategies Implemented 


Emergency Medical Technician, Regional Simulation Center (skills practice), 


: Z 5 EMT licensing exam observations, career navigator, hybrid 
Washburn University instruction 
: . Sim Center (skills practice), workplace skills 
- LPN : : : : 
er oed lire oe curriculum, career navigator, articulation 
agreement, transfer credits 


Information Technology 


Program 
Length 
(Giteyaidit)) 


Program and Grantee 


Credential Types 


Primary Strategies Implemented 


Software Devaopment, Tay Tech® | ee ae er 
Computer Science, Ivy Tech* 24 AAS Transfer Single Articulation Pathways 
; : Certificates in Digital Forensics, Network 
ae aa 21 Penetration, Nene Security, Technical 
‘ Certificate in Cyber Security 
IT Support, Ivy Tech* 10 Certificate in IT Support & IT help desk 
ifi in Routing and Switching, Tech i isi ; i 
Deak dalrsstueuie avy mech ee in ec ie anlar fe Papal ses ‘ani 
Certificates in Linux Administration, Microsoft environment, skills training in virtual 
Server Administration, Ivy Tech* 21 Administration; Technical Certificate in Server environment (data centers) 
Administration 
Certificates in Data Analytics, Database 
Database Management, Ivy Tech* 21 Administration, Technical Certificate in Database 
Mgmt. and Administration 
: Technical Certificate in Informatics 
Informatics, Avy Teeh® Is Certificate in Informatics 
NETI 105, ITSP 135, Ivy Tech* 4 Various industry certifications 
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a Fe E Transportation and Warehousing 


Program 


Program and Grantee Length Credential Types Primary Strategies Implemented 
(Cited) 


Commercial Driver’s License and F F sas Skills training in physical environment, 
Forklift, Cincinnati State 0.25 - coi AU aes Ceruficaton academic support (tutoring), job placement 
Technical College enn daa assistance 


Welding, Multiple Grantees 


Program 
Program and Grantee Length Credential Types Primary Strategies Implemented 
(Gite dity) 
eds ea iy elm 4 AWS Certified Welding Supervisor Equipment & facility for skills practice; 
College AWS Certified Welding Inspector academic supports, accelerated boot camp 
Welding, Chaffey College 18 AWS sequence of credentials Equipment for skills practice 
Welding Technology, Manchester 8 Advanced Manufacturing Welding Technology Equipment for skills practice; stackable 
Community College certificate credentials 
: : Equipment for skills practice in virtual 
Welding, South Central College 4 AWS sequence of credentials snvianmane 
: Welding training facility (skills practice), 
ee Washburn 5 AWS sequence of credentials accelerated curriculum, stackable credentials, 
Credit for Prior Learning (Vets) 
Credential abbreviations Key 
Manufacturing Skill Standards Council (MSSC) Accelerated and enhanced learning strategies 
American Welding Society (AWS) i i i 
National Center for Construction Education & Research (NCCER) Employment connections—work-based learning strategies 


Employment connections—employment services strategies 
* Sample of Ivy Tech certificates and degrees; most programs include associate degrees (not listed) 
Source: Implementation site visits, conducted March and April 2017. 
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Manufacturing. The greatest number of outcomes study programs (14) were manufacturing related. 
Five grantees implemented these programs, which ranged in length from one month to 16 months. 
Resulting credentials were industry specific, including those provided by entities such as the 
Manufacturing Skill Standards Council and the National Center for Construction Education & Research. 
The most common grantee-funded strategies for these types of programs were in the areas of 
connections to employment (equipment and facilities for physical skills practice) and accelerated and 
enhanced learning (creating or enhancing programs within a career pathway). 


Information Technology. The next most common industry was IT. One grantee operated all nine of 
the studied IT programs. Program length ranged from four months for gateway programs to 24 months. 
Grantee-implemented strategies were related to connections to employment (skills practice in both 
physical and virtual settings) and accelerated and enhanced learning (technology-based learning). 


Welding. The outcomes study included five welding programs, operated by five grantees. They ranged 
in length from four to 18 months and resulted in American Welding Society credentials. Grantees used 
funds for strategies in the connection to employment category, primarily purchasing welding equipment 
and expanding welding facilities in order to provide more opportunities for skills practice. 


Other. Less common industries were construction, health care, and transportation/warehousing. One 
grantee implemented three construction programs, ranging in length from three to six months. This 
grantee’s funded strategies were connection to employment (space and equipment for skills training) and 
accelerated and enhanced learning (creation of stackable credentials). One grantee implemented two 
health care programs, ranging in length from five to eight months. The strategies implemented were 
connections to employment (creation of a simulation center for skills practice), accelerated and enhanced 
learning (creation of a new credential), and persistence and completion (funding for a career navigator). 
Finally, one grantee operated a transportation program and a warehousing program. This grantee’s 
strategies were related to connections to employment (equipment purchase and development of a 
training facility to enable hands-on skills practice). 


3.2, ACCELERATED AND ENHANCED LEARNING e(= 
STRATEGIES ra 


This section discusses grantees’ funding of accelerated and enhanced learning strategies. The grantees 
used grant funds to develop and enhance career pathways programs, employed a variety of technology- 
enabled learning techniques, and made adjustments to the structure and timing of teaching and learning 
(modularized, self-paced). Grantees also implemented credit for prior learning or prior learning 
assessments. 


| All grantees designed new or enhanced existing career pathways or 
developed new stackable credentials. 


The Round 4 grant solicitation emphasized career pathways and stackable credentials. The grantees took 
a variety of approaches across different industries. 
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Accelerated and Enhanced 
Learning Strategies Defined 


Career Pathways: Approaches that offer articulated education and training steps between 
occupations within an industry sector, combined with support services, to enable individuals 
to enter and exit the job market at various levels and to advance over time to higher skills, 
recognized credentials, and better jobs with higher pay. 


Technology-Enabled Learning: Strategies that use technology to effectively teach content 
to participants, enable participants to teach themselves, or enable participants to learn from 
one another. Examples include entirely online courses and hybrid courses. 


Prior Learning Assessment, Credit for Prior Learning: A prior learning assessment is 
an evaluation of skills and knowledge acquired outside the classroom for the purpose of 
recognizing competence against a given set of standards, competencies, or learning 
outcomes. Credits are awarded based on this assessment. 


Credit for prior learning is also awarded for non-credit or professional development 
certificates, apprenticeship certificates. 


Structure of Learning: Strategies include modularized learning in which modules are 
conceptualized as self-contained “units”, and self-paced learning. 


Stackable Credentials: A sequence of certificates or credentials that can be accumulated 
over time and move an individual along a career pathway or up a career ladder. 


Examples of career pathways: 


Each college in the South Central College consortium developed a career pathway for its 
advanced manufacturing programs that incorporated the Manufacturing Skill Standards Council's 
Certified Production Technician credential as an entry-level certificate that transferred to more 
advanced diploma and degree programs. The credential comprised of four core courses that colleges 
embedded into several advanced manufacturing pathways in machining and mechatronics. 


Through its grant-funded InTech Center (see Section 3.4), Chaffey College supported participants 
entering and advancing through multiple pathways, particularly in the industrial maintenance 
program, which had heating, ventilation, and air conditioning, industrial-electrical, and industrial- 
mechanical pathways. The three training pathways prepared graduates for employment as an 
industrial maintenance technician. Participants could enter the program through any of the pathways 
and could earn credentials in multiple pathways. 


Examples of stackable credentials: 


The Miami Dade College consortium designed a four-level training for a manufactured 
construction program to introduce participants to the core skills required in the industry. The four 
levels—foundation, basic, advanced, and supervisory—each culminated in a stackable credential.** 
Participants began in the foundation program and proceeded through the program levels in 


31 The supervisory program was not included in the outcomes study because no participant had reached the fourth level at 


the time study enrollment began. 
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sequence. Participants could earn up to seven industry-recognized credentials if they completed all 
four levels. 


e Colleges in the Manchester Community College consortium enhanced their existing advanced 
manufacturing programs to incorporate stackable third-semester credentials. Staff based each 
third-semester certificate on competencies employers identified as necessary for the job. Rather than 
writing a detailed curriculum that could become obsolete in the advanced manufacturing field before 
it was completed, program staff worked backwards from exit competencies identified by employers to 
create learning objectives and develop a course syllabus. For example, Asnuntuck Community College 
developed certificates in additive manufacturing, advanced machining technology, CAD/CAM 
programming, quality inspection, and metal fabrication. When the certificates are approved by the 
Connecticut Board of Regents, program participants will be able to stack credentials by attending 
training for one additional semester. Therefore, according to grant staff, the third-semester 
certificates were more focused and better suited to participants who already completed the core 
advanced manufacturing program or who already had experience working in manufacturing. 


| Eight grantees implemented at least one technology-enabled learning 
strategy. 


Grantee approaches to technology-enabled learning strategies included hybrid learning (where 
instruction is delivered both online and in person), online teaching and learning, and real-time online 
instruction. For example: 


e At Ivy Tech Community College, IT program participants could take courses in a variety of 
formats, including exclusively online. This allowed participants to take courses from any of the 
college's campuses, regardless of whether that course was offered at the physical campus they 
attended. Using grant funds, the college developed and revised curricula for online courses and 
offered new trainings to faculty. According to grant staff, the new and enhanced online 
courses increased participants’ access to training opportunities, especially in smaller and more rural 
areas where there were not enough participants to offer certain courses in person. 


e The South Central College consortium used grant funds to develop +Connect, an online 
mediated telepresence classroom that allowed participants to attend classes remotely. +Connect 
courses were targeted at incumbent workers and those in apprenticeship or dual-training programs 
who did not have time to travel to or attend traditional in-person classes. They created +Connect in 
order to allow participants to work and earn a credential at the same time. In the mediated 
telepresence courses, participants could join an online co-space from any location; instructors could 
share PowerPoint slides, videos, and other media. Participants and instructors used cameras and 
microphones to communicate with each other. 


At least one grantee found that planned technology-enabled learning techniques did not work for 
manufacturing program courses. This grantee explored creating hybrid classes for up to 10 advanced 
manufacturing courses that combined in-person lectures with online instruction. The grantee determined 
that advanced manufacturing courses were not good candidates for hybrid classes because few were 
lecture based and the majority involved extensive lab time. As of the research team’s site visit, no hybrid 
classes had been created. 
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| Five grantees implemented prior learning assessments or awarded credit for 
prior learning. 


Five colleges in the outcomes study implemented new prior learning assessments to assign participants 
credit for their prior educational or work experiences. For example: 


e Washburn University consortium used grant funds to hire a veteran specialist who 
encouraged soon-to-be discharged military service members to apply to any of the programs of 
study. The veteran specialist helped establish credit for prior learning for veterans by aligning 
their skills acquired in the military with Kansas Board of Regents training program requirements. 
Veterans could enter their job code and level of expertise into an online system that then provided 
the number of credits the veteran could receive at programs in the Kansas postsecondary education 
system. During the grant period, the consortium developed veteran-related credit for prior learning 
for the physical therapy assistant program at Washburn University, the surgical technology program 
at Washburn Institute of Technology, and the welding technology program at Garden City Community 
College. 


e Manchester Community College adapted its implementation of credit for prior learning for its 
advanced manufacturing programs after determining the field does not easily lend itself to prior 
learning assessments. The two primary assessment methods were problematic. Evaluation of a 
portfolio of work to demonstrate skills was not possible for manufacturing. Administration of a skills 
test was possible but would require instructors to develop their own assessments because there was 
no existing standard test of manufacturing skills and knowledge. According to grant staff, some 
instructors were open to implementing their own skills assessment but noted that there were few 
requests from participants. One program director noted that when participants did request an 
assessment, he allowed them to spend more class time practicing skills on machines and less time in 
lectures. 


e South Central College grant staff worked with the college’s Continuing Education and Customized 
Training department to measure competencies gained by participants who completed the non-credit 
certified production technician courses. If a course participant enrolled in a more advanced machining 
or mechatronics course, college staff assessed the participant’s competencies to award credit for 
prior learning. According to grant staff, other colleges in the consortium developed similar 
arrangements. 


| Most grantees implemented a new learning structure for their training 
programs. 


Modularized learning, in particular, was common among manufacturing programs, especially those 
implementing industry-recognized National Center for Construction Education & Research and 
Manufacturing Skill Standards Council certifications. For example: 


e In the foundation program in the Miami Dade College consortium’s training for manufactured 
construction sequence, participants progressed through performance evaluations associated with 
each module. In each evaluation, participants had to demonstrate their competence in a specific skill. 
For example, at the conclusion of one module, participants had to show they could effectively use a 
handsaw to cut a wood 2x4. The foundation modules included hand tools, power tools, plan reading, 
communication, and employability skills. At the end of the modules, participants earned a National 
Center for Construction Education & Research Core certification. 
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e Two grantees used self-paced learning techniques in their grant-funded welding training 
programs. Instructors in Bossier Parish Community College's advanced welding program taught 
participants by using hands-on, self-paced learning techniques in the welding booth, a skills practice 
physical environment. Similarly, in the Washburn University consortium, Garden City Community 
College redesigned its welding program curriculum to include self-paced learning such that 
participants could prove competency in a weld and then progress to a more complicated one. 


3.3. PERSISTENCE AND COMPLETION STRATEGIES eRe 
rs 


Five grantees used funds to hire and support staff who provided enhanced academic and 

nonacademic supports. In several cases, the same position (e.g., navigator, retention and completion 
specialist) provided both academic supports, such as tutoring, advising, and monitoring academic 
progress, and nonacademic supports, such as case management, goal setting, and connections to 
supportive services. Grantees implemented contextualized learning (e.g., math built into welding training 
programs), competency-based training (advancement to the next module or course based on skills 
attainment rather than seat time), and transfer and articulation policies. 


| Grantees used grant funds to provide a range of academic supports. 
The nine outcomes study grantees implemented one- 
on-one advising, tutoring, and team teaching to 
support academic success. For example: 


Persistence and Completion 
Strategies Defined 


Ivy Tech Communi HT developed a Enhe Academic Sup t 
sa ALi te lec aaa Examples include tutoring, academic 


advising (e.g., course planning), and 
academic preparation and course 
support (e.g., study skills). 


virtual career pathway advising tool for its 
School of Information Technology. Because the 
programs of study in the school were closely 
related and had multiple overlapping course 
requirements, faculty found that program 
participants were taking multiple classes, but in 
combinations that did not meet the completion 
requirements of any program of study. This 
advising tool helped program participants choose a 
program of study and understand the jobs and manage time, how to work well within a 
associated salaries for which they would be team, how to communicate, and how to 
trained. After the program participant chose a manage personal finances. 

program, the tool mapped out course 
requirements, when those courses were offered, sequenced prerequisites, and recommended against 
certain “killer combinations” of difficult courses taken in the same semester. 


3: Examples 
include case management, goal setting, 
referrals to supportive services, financial 
aid advising, and career counseling. 


: Examples include classes or 
advising on how to take tests, how to 


e As part of grant activities, colleges in the South Central College consortium implemented 
“intrusive advising,” one-on-one academic and career advising that all program participants 
enrolled in a grant-funded program received. According to grant staff, the extent of advising was 
more intense and individualized than the traditional academic advising offered by the colleges. 
Intrusive advising included working with participants to detect and resolve persistence issues by 
monitoring program attendance, speaking with instructors, and interpreting the results of academic 
assessments. 
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e The Washburn University consortium allied health career navigator conducted in-class 
presentations to review each program’s requirements, length, and career pathway; assessed 
participants’ academic skills and provided referrals to supports as needed to help them persist in and 
complete the program; and offered one-on-one career and educational advising. The career 
navigator proactively distributed a list of commonly needed supports, such as assistance with study 
skills or test anxiety. This navigator worked with participants in two outcomes study programs— 
licensed practical nurse and emergency medical technician. 


e One college in the Chaffey College consortium used grant funding to hire three peer mentors 
for its manufacturing program to serve as lab assistants, help instruct program participants, help run 
the tool room, and ensure that participants used lab equipment safely. According to grant staff, 
because peer mentors helped run the training equipment and labs, instructors had more time to 
spend on other activities, such as creating strong partnerships with employers, developing 
curriculum, and instructing participants. 


Some grantees provided tutoring support, primarily related to basic skills: 


e Cincinnati State Technical and Community College provided academic support and 
tutoring through a partnership with a community-based organization. There were no basic skills 
requirements at the college's forklift training program. However, when participants demonstrated low 
basic skills, the college arranged one-on-one basic skills training with a tutor at the organization. 


e One college in the Miami Dade College consortium integrated academic support staff into its 
training for manufactured construction programs. An instructor from the Adult Education 
department attended the first few semesters of the foundation course to provide help with reading 
and math. This instructor also provided tutoring as requested by participants. 


One grantee integrated a team-teaching approach, developed under a prior round grant, into the two 
new Round 4 programs included in the outcomes study. 


e Delgado Community College used grant funds to implement team teaching in the TECH 101 
and Core Plus programs. The teaching team included the occupational skills instructor and a basic 
skills instructor. Modeled after the Washington State Integrated Basic and Education Skills Training 
program (I-BEST), the basic skills instructor was in class at least a quarter of the time and provided 
individual assistance to participants who needed additional support and supplemental instruction on 
foundational academic skills (e.g., basic math, reading, and writing). At times, the basic skills 
instructor participated in class discussions by asking the occupational instructor probing questions to 
clarify material that appeared to confuse participants. 


| Grantees implemented a range of nonacademic supports. 


Grant-funded staff such as program navigators and recruitment, retention, and completion specialists 
provided nonacademic supports, either one-on-one or in classes. In some instances, the same staff 
provided academic and nonacademic supports. For example: 


e At Delgado Community College, two program navigators provided case management to 
promote participant retention in and completion of programs. Navigators worked with each 
participant to create an Individual Service Strategy, a standardized form covering educational history, 
work history, transportation plans, history of involvement in the criminal justice system, supportive 
service needs (e.g., childcare, counseling, housing, legal assistance), professional goals, and an 
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action plan. The navigators helped participants with various logistics (e.g., registering for courses) 
and provided career advising. 


e All colleges in the Miami Dade College consortium had at least one grant-funded recruitment, 
retention, and completion specialist. The specialists met with participants one-on-one to develop 
an individual learning plan that mapped participants’ goals and identified ways in which the 
specialists could assist with goal achievement. For example, a plan could explain how the specialist 
would connect the participant to supportive services, such as childcare or transportation assistance. 
The grantee encouraged but did not require participants to meet with a specialist. 


e In addition to academic and career supports, the Washburn University consortium career 
navigator assessed participants’ social service needs and provided referrals to supports to help them 
persist in and complete the program. The career navigator’s list of commonly needed participant 
supports included information about a food pantry. 


Examples of Staff Providing 
Academic and Nonacademic Supports 


ram Navigators (Delgado Community College): Provided case management and 
served as a resource fai maricpante iKe) rOniele retention aad (elo) na}8)(=10(0) 9 


Career Navigators (Wash University consortit Provided information about 
program requirements, efrcred one-on-one career rand Sed advising, assessed 


sae social service needs, and provided referrals to supports. 


int ve Ad (South Ce 1 Oe) | (=e ortium): Provided one-on-one 
Beenie onl career FaaUEInG. Worked with paerare to ee and resolve persistence 
issues. 


sareer Coaches (Bossier Parish Community College): Provided academic advising, set 
goals with eeicnanie ani eee with resume aceae eee and interview preparation. 


Finally, one grantee implemented training to help participants prepare for work in a multi-disciplinary 
environment: 


e Inresponse to local employers’ requests that all allied health program participants be prepared to 
work in a multi-disciplinary environment, the Washburn University consortium purchased and 
implemented TeamSTEPPS (Team Strategies & Tools to Enhance Performance and Patient Safety) 
training. TeamSTEPPS taught health care communication strategies and helped participants 
understand the roles and responsibilities of various health care providers with whom they may work 
after completing training. 
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| Grantees also implemented strategies to award credit for non-credit course 
completion. 


Examples of articulation from non-credit to credit: 


e The Miami Dade College consortium developed agreements within their own colleges that 
allowed for the grant-funded training for manufactured construction programs and credentials earned 
to count for credit in more advanced degree programs. Such credits were awarded even though 
the programs were not for credit. For example, at Miami Dade College, the sequence of four 
programs together counted for 15 to 17 credits in the college’s construction and architecture degree 
programs. At Seminole State College, each credential that participants received through the programs 
counted for credit in the college’s construction management associate’s degree program. Participants 
could earn up to 11 credits if they received credentials associated with all four program levels. 


Finally, one grantee implemented an articulation agreement to help graduates of a for-credit program 
transfer credit to another institution of higher learning for a subsequent degree: 


e As part of the Washburn University consortium's grant activities, Washburn Institute of 
Technology executed a new articulation agreement with Washburn University. The agreement 
allowed practical nursing participants to bring 18 credits to Washburn University’s Bachelor of Science 
in nursing program. An earlier version of the agreement had capped the number of credits at 12. The 
articulation agreement allowed practical nursing program completers to take one fewer course each 
semester while training for their bachelor’s, which gave them flexibility to work while attending 
school. According to instructors, this type of articulation agreement was relatively rare in nursing. 
Many programs articulate between an associate’s and bachelor’s degrees in nursing but they do not 
connect with a practical nursing program. 


3.4. CONNECTIONS TO EMPLOYMENT STRATEGIES 


Connections to employment strategies include work-based learning strategies (e.g., skills practice 
in physical or virtual physical environments and in-program work experience) and employment-related 
services (e.g., job placement services and interviewing practice). 


All grantees implemented work-based learning strategies, primarily skills practice. Seven grantees offered 
in-program work experience. Grantees also implemented a number of job search assistance activities 
(e.g., help with resume development, interview preparation, connections to specific job opportunities). 
Three outcomes study grantees offered internship opportunities. 
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| All nine grantees implemented Connections to 
skills practice strategies. Employment Strategies 


: Examples include 


in either a physical or virtual setting. Colleges e Practicing skills through simulations in a 
implemented simulated learning techniques in virtual setting 

diverse fields, ranging from manufacturing to health e Training in a work-like physical 

care to information technology. For example: environment 


e In-program work experience 
(e.g., internships, clinical placements, 
r-].9) 9) =1a) W(ec=1-) 011 8)<)) 


Ivy Tech Community College used a large 
portion of the grant for new or updated 
computer labs and physical and virtual data 
centers. The college bought new computers 


: Examples include 


to upgrade old labs, to expand labs at e Job developers to identify potential 
campuses with high numbers of IT program openings 

parneipanits ale Creare seme (operat e One-on-one assistance or workshops to 
campuses that previously did not have any. prepare for job search, including 
According to grant staff, these investments assistance looking for jobs, creating 
helped ensure that all program participants had resumes, practicing interview skills, 
access to computer labs, and they allowed ii It aYefe)6) mn (0) 9)r-]9)0)|(et=) u(e) as 


instructors to increase the amount of time spent 
in class providing hands-on skill-building lessons. The college also built both physical and virtual 
data centers with servers and networks so program participants could practice building and 
maintaining networks. 


The Washburn University consortium used grant funds to develop the Regional Simulation 
Center (Sim Center), a state-of-the-art center in which allied health profession participants could 
practice their diagnostic and treatment skills and could work in an integrated, “real life” setting. For 
example, a participant could practice helping a patient recovering from hip surgery navigate the 
restroom, key in the requisite codes to dispense medication from a locked medicine cabinet, or 
provide instrument assistance during a surgery. The Sim Center was designed to look like a hospital, 
with an operating room, nurses’ station, and other hospital settings. Instructors in the consortium’s 
licensed practical nurse program used the facility four times per semester to enable program 
participants to practice skills via simulations. 


The Chaffey College consortium’'s major investment was its InTech Center, which supported 
training in industrial maintenance, pre-engineering, solid works, and 3-D manufacturing. The Center 
has multiple skills practice workstations, such as welding booths, a warehouse, and conveyor belts, to 
provide hands-on training to program participants. The grantee used funds to purchase equipment, 
such as a forklift simulator. The grantee also funded equipment that enhanced training delivery, such 
as a welding simulator. 


Each college in the Miami Dade College consortium had a Transformative Learning Center, a 
high-tech woodshop and lab. The Transformative Learning Centers varied in size across colleges. 
Staff at all three outcomes study colleges reported the labs gave program participants the 
opportunity to learn concepts firsthand, which in turn helped them learn the material faster. 
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Simulation opportunities were particularly common among welding programs. For example: 


Bossier Parish Community College used grant funds to purchase welding booths and a 
welding simulator. The welding simulator allowed participants to learn proper welding techniques 
in a virtual setting without the need for consumables, such as metal and gas. The simulator provided 
the look and feel of a real welding machine. Participants used a welding gun to perform welds in a 
virtual environment displayed through a screen in their helmet. They could interact with workpieces 
in the virtual environment using their welding gun. Participants and instructors received real-time 
feedback from the simulation software. 


| Seven grantees provided job search services. 


A variety of staff provided job search services, including instructors, dedicated career coaches, job 
placement specialists, and career developers. For example: 


In Cincinnati State Technical and Community College's forklift program, job placement 
specialists brought employers on-site to conduct mock interviews with participants and offer job 
search advice. These employers would use the time on-site to determine which participants they 
wanted to hire. Participants who were not hired in an employer's on-site interview worked with the 
job placement specialist to refine their resume and submit additional job applications. 


At Bossier Parish Community College, a grant-funded career coach helped participants develop 
their resumes and prepare for interviews. The career coach also provided career preparation support 
services by teaching participants employability skills and offering personal branding workshops. The 
workshops focused on helping program participants learn how to market themselves as they entered 
the job market. 


At Delgado Community College, a career developer managed relationships with employers, 
connecting participants to employment and interview opportunities. The developer helped 
participants with their resumes, interviewing skills, and job search strategies. The grantee also 
implemented job readiness workshops, both in-class and standalone, that covered topics such as 
professional skills, job search strategies, teamwork, and conflict resolution. 


| Internships were the most common work-based learning activities among 
grantees. 


Three grantees offered internship opportunities of varying length and formality. For example: 


The Manchester Community College consortium's advanced manufacturing program included a 
combination of lecture courses, lab work, and a paid internship. Participants had to interview for 
internships, but program staff provided assistance identifying potential employers. At Manchester 
Community College, participants were required to spend the last eight weeks of their advanced 
manufacturing program in a paid internship. According to grant staff, the internship often 
translated into a full-time job. At Naugatuck Valley Community College, participants participated in 
paid internships that were eight weeks long, two days a week. Participants were required to have a 
recommendation from their full-time instructor and good attendance in order to qualify. Around 85 percent 
of interns transitioned to a full-time job with that company. Before the internship period, the college 
alerted all companies on its list, asking those that had a need for an intern to respond. 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1 28 


CHAPTER 3: IMPLEMENTATION FINDINGS 


One grantee used clinical placements to help program participants learn and further hone skills: 


e Inthe Washburn University consortium, practical nursing participants at Washburn Institute of 
Technology attended rotations at several major local health care providers. The rotations showed 
participants the types of institutions where they could seek jobs, including a hospital, a nursing 
home, and a rehabilitation center. In the college’s emergency medical technician program, 
participants were required to log 20 clinical hours—12 hours in an emergency room department 
and eight in ambulance ride-alongs. 


A few grantees used grant funds to plan and develop apprenticeship opportunities. For example: 


e At Bossier Parish Community College, a grant-funded apprenticeship coordinator was 
responsible for identifying and developing apprenticeship opportunities with employers that might 
benefit from apprenticeship and learning about DOL’s Registered Apprenticeship College Consortium. 
The coordinator’s goal was for the college to become a Registered Apprenticeship sponsor. As of the 
research team’s site visit, the college had engaged in serious conversations with two employers about 
offering Registered Apprenticeships. 
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4. Characteristics of Participants 
at Program Entry, Training 
Duration and Service Receipt, 
and Participant Outcomes a 
Year Later 


This chapter presents characteristics of and outcomes for all participants who were enrolled in the 
34 programs included in the outcomes study. The chapter first describes the socioeconomic status, 
demographics, and educational and career aspirations of study participants at program entry. It then 
describes training duration and participants’ service receipt in each service area (accelerated and 
enhanced learning, persistence and completion, and connections to employment). 


Finally, the chapter describes a number of Theory of Change “intermediate outcomes” measured about 
15 months after program entry. Those intermediate outcomes are related to training progress, 
employment, earnings, and public assistance receipt. (Outcomes for subgroups are presented in 
Chapter 5.)32 


The chapter addresses outcomes study research questions 1 through 5: 


What were the characteristics of study participants? 


In which features of training programs and services did study participants engage? What 
college and partner (e.g., workforce agency) services did they receive? What were their 
assessments of the services they received? 


What education outcomes did study participants in short-term training programs 
achieve? Did they see themselves on a career pathway? What were their plans for future 
training? What was the expected timeline? 


Did participants obtain employment? If so, was this employment in the occupation for 
which they trained? What were their earnings? How did it compare to their earnings 
Prior to entering their training programs? What were the characteristics of their jobs? 


Did household income change? Did study participants receive public assistance benefits 
less frequently at follow-up than at program entry? 


32 Detailed tables underlying the figures presented in this chapter are available in Appendix G. 
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Key Takeaways: 
Characteristics of Participants at Program Entry 


The study participants were mostly male, mostly never married, and diverse in terms of 
race/ethnicity and age. Most participants (53 percent) had at least some postsecondary 


education at program entry. 


The share of participants receiving public assistance was low: 10 percent or less. 


Key Takeaways: 
Outcomes a Year after Program Entry 


Participants were generally satisfied with the grant-funded services they received. 


The majority of participants reported they completed their training programs. A majority 
planned to return to college. 


The vast majority of participants no longer enrolled in training at follow-up had been 
employed since exiting their program. Nearly half (44 percent) of program completers 
were employed in jobs related to their training. 


Average quarterly earnings increased by $2,272 from three quarters before program entry 


to five quarters after program entry. 


A majority of participants who were no longer enrolled in their programs were employed 
full-time in positions with benefits. 


Greater percentages of program completers than non-completers were employed in jobs 
related to their programs and in those with benefits. 


Participants’ household income increased and the percentage living in poverty decreased. 


Participants’ receipt of public assistance benefits remained stable. 


4.1. CHARACTERISTICS OF PARTICIPANTS AT PROGRAM ENTRY 


The participants in the 34 programs studied were diverse in terms of demographic, socioeconomic, and 
economic characteristics. They also had varying educational and employment histories. This section 
describes the characteristics of the participants based on the data captured in the Baseline Information 
Form. 


| Most participants were male and most were single, but they were otherwise 
diverse demographically. 


Eighty-five (85) percent of participants were male, reflecting the large share of programs in the 

manufacturing sector, jobs that have traditionally been occupied by men.*? Several colleges that offered 
manufacturing programs (e.g., Chaffey College, Manchester Community College, South Central College) 
intended to recruit women. However, staff reported difficulty recruiting women despite outreach efforts. 


33 See, for example: 3 // Ww 
(last accessed 15 June 2020). 
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The majority of participants were single. As 
Exhibit 4-1 shows, 56 percent were not living 
with a spouse or partner, nor did they live with 
children. Twenty-nine (29) percent lived with a 
spouse or partner (either with or without 
children). The remainder lived with children but 
not a spouse or partner. 


Nearly one-third (29 percent) of participants lived 
rent-free at program entry. Fifty-three (53) 
percent rented their own place or contributed to 
rent at a friend of family member's place while 

18 percent owned the place where they lived (not 
shown). 


Participants were diverse in terms of other 
demographic characteristics. Forty (40) percent 
were traditional college age—that is, 24 years or 
younger (Exhibit 4-2). Those older than 
traditional college students were almost evenly 
split between ages 25 to 34 and 35 or older. 


Exhibit 4-1. Participants’ Family Structure at 
Program Entry 


Living with 
spouse/partner not 
living with children, 

13% 


Not living with 
spouse/partner, living 
with children, 14% 


Not living with 
spouse/partner or 
children, 56% 


Source: Baseline Information Form 
Note: Sample includes all study participants (WV = 2.767). 


Participants were racially and ethnically diverse, as well. Almost half (48 percent) were white (non- 
Hispanic), and more than one fifth each were Hispanic (any race) and black (non-Hispanic) 
(Exhibit 4-3). Four percent of participants were Asian (non-Hispanic). 


Exhibit 4-2. Age at Program Entry 


20 years or 
younger, 23% 
35 years or older, 
29% 


25 to 34 years, 
30% 


Source: Baseline Information Form 


Note: Sample includes all study participants (VV = 2,767) 
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Exhibit 4-3. Race and Ethnicity 


Other, including multi race 
r (Non-Hispanic), 4% 


Asian 
(Non-Hispanic), 4% 


Hispanic, any 
race, 22% 
White 
(Non-Hispanic), 


48% 


Source: Baseline Information Form 
Note: Sample includes all study participants (VV = 2,767). 
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As Exhibit 4-4 shows, most participants at 
program entry had at least some 
postsecondary education (53 percent), most 
commonly some college but no degree 

(32 percent). The largest share of participants 
(43 percent) had a high school diploma or 
GED. Few participants were at either end of 
the education spectrum: 4 percent had less 
than high school and 8 percent had a 
bachelor’s degree or more. 


| The majority of participants 
were working at program 
entry or had worked within 
the past year. 


As Exhibit 4-5 shows, more than half 

(56 percent) of participants were employed at 
program entry. Thirty (30) percent were not 
working at the time but had worked within 
the previous 12 months. Less than 15 percent 
were long-term unemployed or had never 
worked. 


A larger share of participants—77 percent— 
expected to work in the months following 
program entry, either full time (47 percent) or 
part-time (30 percent). About one fifth did 
not plan to work any hours. 


| Manufacturing was the most 
common industry reported by 
participants employed at 
program entry and by those 
unemployed but with recent 
work experience. 


As shown in Exhibit 4-6, similar proportions 
of participants employed at program entry 
and those unemployed but with work 
experience reported employment in 
manufacturing (18 percent and 17 percent, 
respectively). Beyond manufacturing, industry 
differed by employment status. Employed 
participants reported the next most common 
industries were retail trade (15 percent), 
accommodation/food service (11 percent) 
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Exhibit 4-4. Educational Attainment at Program 
Entry 


Less than high school, 4% 


Associate's 


degree, 7% Bachelor's 


degree or 
more, 8% 


High school or 
GED, 43% 


Some college but no 
degree, 32% 


Technical or vocational degree, 6% 


Exhibit 4-5. Employment Status at Program Entry 


Has never worked, 4% 


Has not 
worked for 
12 or more 

months, 10% 


Working at one 
or more jobs, 
56 % 


Not currently working, 


but worked at one or 
more jobs in the past 
12 months, 
30% 


Source: Baseline Information Form 
Note: Sample includes all study participants (WV = 2 767) 
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construction (10 percent), and other services (10 percent). Unemployed participants with work 
experience reported the next most common previous industries were construction (17 percent) and 
transportation/warehousing (13 percent). 


Exhibit 4-6. Prior Industry of Participants by Employment Status at Program Entry 


nn 17% 
ESS ee 


Manufacturing 


en 17% 
Co 10% 


Construction 


_ ee 13° 
Transportation and Warehousing SSS 13% 
(2) 


—__, 11. 
Other S 
OSM (EE 10°, 


| og EE 10% 
A dat d Food S 
Ccomodaton and Food SSCS EE 11° 


‘rece 7° 
Reto 20 15%. 


Ce 4% 
Cee 8% 


Health Care and Social Assistance 


0, 
Professional, Scientific, and Technical Services a ms 


0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20% 


ACurrently Unemployed Participants with Prior Work History §@Currently Employed Participants 


Source: Baseline Information Form 

Note: Sample in the analysis includes participants employed at enrollment (VV = 1,563) and unemployed participants 
with prior work history at enrollment (VV = 827). Participants with no prior work history are excluded, as are 
participants with rare industry backgrounds. 
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The average hourly wage for participants employed at program entry was $14.09, lower than the last 
hourly wage among those unemployed at program entry ($14.55) (not shown). This difference may 
reflect differences in industry types. As noted above, beyond manufacturing, currently employed 
participants reported retail and accommodation/food services as common industries, whereas 
unemployed participants with work experience reported construction and transportation/warehousing, 
industries that generally pay higher wages. ** 


Exhibit 4-7 shows which reason participants Exhibit 4-7. “Most Important” Motive for 
reported as “most important” for enrolling in Enrollment 


their training programs. The most common 
reason cited was career advancement 

(29 percent), followed by career change Personal 
(25 percent). One fifth cited finding work as setae! 
their reason. 


Other, 3% 


Educational 
The majority of participants had no prior advancement, é 
experience in the target industry (60 percent) 14% coeur 
(not shown). About one third reported a year 29% 


or more of experience in the target industry; 
the remainder had less than a year of 


experience. Find work, 


20% 
| Most participants were low Career change, 
income but few received 25% 

public assistance benefits. 


Thirty (30) percent of participants reported 

their family income was less than $15,000 

the year prior to program entry, and slightly 

fewer (27 percent) reported income between Source: Baseline Information Form 

$15,000 and $29,999 (not shown). The 

average family income was $33,526. Nearly 43 percent of participants had family incomes below the 
poverty level. ?° 


As Exhibit 4-8 shows, few participants reported receiving public assistance at program entry. The most 
common benefit reported was SNAP (11 percent of participants). Very few participants received Trade 
Readjustment Allowances (TRA) or TANF (2 percent each). 


34 The May 2019 U.S. Bureau of Labor Statistics occupational and employment statistics report that the mean hourly wage 
is $20.06 for construction laborers and $18.23 for the transportation and moving materials industry. In contrast, the 
mean hourly wage is $13.27 for retail sales workers and $12.82 for food preparation and serving related occupations 
(https://www.bls.gov/oes/current/oes_nat.htm#35-0000). 

35 Poverty status was calculated from reported family income and family size, using 2017 federal poverty guidelines 
(https://aspe.hhs.gov/2017-poverty-guidelines). For example, the poverty level for a family of three was $20,420. 
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Exhibit 4-8. Public Assistance Benefit Receipt at Program Entry 
14% 
12% 11% 
10% 
8% 
6% 5% 
4% 


2% 2% 


Received SNAP Received UI Received TRA Received TANF 


2% 


0% 


Source: Baseline Information Form 
Note: Sample includes all study participants (VV = 2,767). 


4.2. TRAINING DURATION 


All study participants received some training due to the timing of study enrollment, which occurred on the 
first day of class or shortly thereafter. This section reports the months of training, full-time-equivalent 
(FTE) months of training, and completion of at least six FTE months of training. Exhibit 4-9 shows the 
average values, both in total and by completion status. On average, participants studied for 7.4 months. 
However, they were not all full-time students. Treating 12 hours per week in class as full-time, the 
average number of FTE months of training was 6.2.6 


A recent evaluation of a grant-funded short-term health care training programs found that the programs 
succeeded at increasing credential receipt among study participants who could access the programs but 
that credential receipt had not translated into detectable earnings impacts three years post study entry 
(Peck et al. 2019). Most of the credentials earned required a few months of study. The evaluation team 
speculated that persisting through at least six months of study might be more likely to lead to higher 
earnings and therefore declared six months of training to be a threshold of interest for this study of 
TAACCCT.*” Forty-four percent of the full sample reached or exceeded this threshold. 


Training duration was naturally much longer for participants still enrolled in their original program at 
follow-up a year after program entry. Among these participants, the average number of enrolled months 
was 14.0, the average number of FTE months was 10.7, and 83 percent reached the threshold of six FTE 


36 The assumption here is that there are two hours of class preparation for every hour in class, so 12 hours per week in 
class translates of 36 hours of study per week. 

37 This decision was also made for a parallel evaluation of health care training programs (Judkins, Klerman and Locke 
2020). Even this might not be a high enough threshold since Dadgar and Trimble (2015) found that short-term 
credentials (those that require less than a year of study) do not increase the earnings of people who receive them, but it 
does seem credible that 6 months of training are more likely to result in earnings growth than just a handful of months. 
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months. For those who finished the program or who stopped attending without finishing at follow-up, the 
duration of training was much shorter. Finishers studied 6.2 months overall and 5.5 FTE months; 39 
percent achieved the six FTE month threshold. 


Exhibit 4-9. Training Duration 


16 
14.0 
14 
12 10.7 
10 
Pe 
S : 6.2 _ 6.2 
> 5.6 BS 
6 4.6 
4 
2 
0 
Still enrolled in required Stopped without Finished classes All partidpants 
classes completing 


A Months of training @ Full-time equivalent months of training 


Source: Short-term participant follow-up survey 
Note: Sample includes all survey respondents (WV = 2,211). 


4.3. SERVICE RECEIPT 


In addition to classroom training, grantees provided an array of supplemental services to support 
participant completion of training. These include services that accelerate or enhance learning, support 
persistence and completion, and/or provide connections to employment with the aim of improving 
education and employment outcomes for participants. This section focuses on select academic and 
nonacademic supports and services provided by grantees, as reported by participants on the short-term 
follow-up survey. The services included in this section are not exhaustive of all services that participants 
may have received or grantee colleges may have offered. Rather, services included are those highlighted 
in the TAACCCT theory of change presented in Chapter 2 and could be self-reported by participants. 


| Most participants reported their program was part of a career pathway; a 
minority reported engagement in other accelerated and enhanced learning 
strategies. 


The most common accelerated and enhanced learning strategies reported include career pathways that 
guide advancement to subsequent programs and credentials, transfer credits, and credits for prior 
learning. Sixty-nine (69) percent of participants reported there was a recommended program that led to 
a more advanced credential (not shown). This finding aligns with the implementation study; all nine 
outcomes study grantees reported that they were implementing new or enhancing existing career 
pathways programs. 


Although most grantees in the study awarded credit for prior learning, only 4 percent of participants 
reported receipt of such credit, and the average number of credits received was one. A possible factor is 
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the predominance of manufacturing programs in the study. The most common methods of assessing 
prior learning are review of a portfolio of work or a skills test. Advanced manufacturing does not lend 
itself to portfolio review because participants cannot assemble examples of their work. And, according to 
program staff, there is no standard skills assessment for advanced manufacturing. 


About one fifth of the sample (21 percent) received transfer credits; among those who received them, 
the average was six credits (not shown). 


| Persistence and completion strategies reported included a range of academic 
and nonacademic supports, most commonly academic advising. Many 
reported taking classes in other skills, with varying topics. 


Nearly half of participants (47 percent) received academic advising (not shown). This may reflect the 
intensive advising designed and implemented by several grantees, including Delgado Community 
College’s navigation feature?® and South Central College’s intrusive advising. *? 


As Exhibit 4-10 shows, the most commonly reported persistence and completion strategies not related 
to advising were providing opportunities for participants to learn other skills: acting professionally 

(29 percent), critical thinking and problem solving (27 percent), communicating well (26 percent), and 
working in groups (25 percent). The least common persistence and completion strategies received 
included classes on handling parenting and other family responsibilities (8 percent) and managing 
finances (10 percent). 


Exhibit 4-10. Topics Covered in Classes that Taught Skills Other than Occupational Skills 


Acting professional 70°), 
Critical thinking and problem sol VinQG a 27 9/, 
Cnnuiictin Vell a 26°, 
Working in (OUST 250), 
Study ski|S es 24 °/, 
Cree DanC A 270, 
Tine nner $s) )°/, 
Staying Motivated LL 27°, 
Ib scr es 2 | °/, 
Finding help with life pole. S a 19°; 
Financial aid for schoo)|l = | 6°, 
Managing stress and angry i 7 1 °/, 
Managing francs 7 ()°/, 
Handling family responsbiitc __ §°%, 


0% 5% 10% 15% 20% 25% 30% 


Source: Short-term participant follow-up survey 
Note: Sample includes all survey respondents (WV = 2,211). 


38 At Delgado Community College, two program navigators provided participants with intensive case management, 
developed individual service strategies, and acted as a resource to promote retention and completion. 


39 Intrusive advising included working with participants to detect and resolve persistence issues by monitoring their 
attendance, speaking with instructors, and interpreting the results of academic assessments. 
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| Nearly one third of participants received financial aid advising to help them to 
persist in and complete their program. The most common source of funding 
for training was the participant’s own earnings or those of a spouse. 


Almost a third of participants (31 percent) received financial aid advising (not shown). Several colleges, 
including colleges in the Chaffey College and South Central College consortiums, provided referrals to 
financial aid advising. 


As Exhibit 4-11 shows, half of participants (50 percent) funded their training at least partially with their 
own savings or those of a spouse. The average amount of funds used was $2,335 among those using 
any of their own funds. This varied by enrollment status, with those still enrolled at the time of follow-up 
reporting more funds used ($2,928) than those who completed their program ($2,334) (not shown). This 
could reflect that participants still enrolled were in longer-term and more expensive programs, such as 
associate’s degrees. Those who left their program before completion reported spending an average of 
$1,748. 


A quarter of participants received financial assistance from a parent or other family member. Loans were 
slightly less common (21 percent); among those who took out loans, the average amount borrowed was 
$3,837 (not shown). Currently enrolled participants and those who completed reported similar amounts 
of borrowed funds (about $4,000), while those who left their program prior to completion reported 
$3,114 in loans. 


Almost 40 percent received grants from the government such as Pell grants. Few used veterans’ benefits 
(5 percent) or TRA benefits (8 percent) to fund training or received assistance from their employers 
(9 percent). 


Exhibit 4-11. Sources of Financial Assistance for Training Programs 


Funds from participant’s and/or spouse's earnings SS eee 
or savings 
Grant from the government i 39% 
Funds froma parent or other family member i 25% 
Loans in own name or name of a family member i 21% 
Other funding source i 16% 
Scholarship i 14% 
Financial support from employer HE 9% 
TAA benefits HE 8% 
Veteran's benefits HI 5% 


0% 10% 20% 30% 40% 50% 60% 


Source: Short-term participant follow-up survey 
Note: Sample includes all survey respondents (WV = 2,211). 
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| Large percentages of participants engaged in connections to employment 
strategies, most commonly work simulations. 


As shown in Exhibit 4-12, more than 80 percent of participants reported engaging in work simulations. 
This reflects the large grant-funded simulation centers (the Chaffey College consortium’s InTech Center 
and the Washburn University consortium’s Sim Center at Washburn Institute of Technology) as well as 
the numerous information technology, welding, machining, construction, and other labs funded or 
enhanced with the TAACCCT grant. About half of participants (48 percent) reported engaging in 
computer or other simulations where they practiced skills in virtual settings. 


Fifty-one (51) percent of participants reported arranged visits from or to employers to learn about their 
work and types of jobs available. More than one third (35 percent) attended classes taught by instructors 
from a local employer or classes offered on-site at a local employer. 


Additionally, 57 percent of participants reported some type of direct work experience as part of their 
program (not shown).*? The most common type of work study was a work study job (27 percent), 
followed by an internship (25 percent) and an apprenticeship (22 percent). Almost 30 percent reported 
participation in some other type of work study. 


Exhibit 4-12. Participation in Various Types of Work-based Learning Experience 


Work-like physical environments to practice skills SI 81°. 
Virtual workplace simulations to practice skills [i 40° 
Offered arranged visits from employer/learning about —————— oc ae 
employers ’ 


Offered class taught by instructors from local 3°. 
employer/class 7 
Other work experience NN 30% 
Work study job i 27% 
Internship [x 25% 
Apprenticeship [i 22% 


Clinical experience or practicum [a 22% 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 


Source: Short-term participant follow-up survey 
Note: Sample includes all survey respondents (WV = 2,211). 


40 Participants were asked, “As part of your studies for [PROGRAM NAME] at [SCHOOL NAME], have you been offered any 
of the following opportunities for direct experiences with occupations related to your studies or career goals?” They were 
then presented with each of the opportunities presented in Exhibit 4-12. Fifty-seven (57) percent reported that they 
received direct work experience, either as a word study job, internship, apprenticeship, clinical experience or practicum, 
or other work experience. 
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| About 40 percent of participants received job search or placement assistance. 


Forty (40) percent of participants received job search or placement assistance. As shown in 

Exhibit 4-13, the most common types of assistance received were help looking for a job (33 percent), 
help creating or editing a resume (33 percent), and help finding specific job leads (30 percent). These 
were the types of services implemented by grant-funded career coaches at Bossier Parish Community 
College, job placement specialists at Cincinnati State Technical and Community College, career 
developers at Delgado Community College, and other staff at the grantees. 


Exhibit 4-13. Receipt of Various Types of Job Search and Placement Assistance Services 
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Source: Short-term participant follow-up survey 
Note: Sample includes all survey respondents (WV = 2,211). 


| Nearly all participants who received advising and employment services did 
so from colleges rather than partners. 


Exhibit 4-14 summarizes which organizations provided various services to participants. Among 
participants who received academic advising, 90 percent reported they received it only from the college. 
Eighty-seven (87) percent of participants who received financial advising and 85 percent who received 
career counseling received those services only from the college. Some participants said they received 
services from both the college and another organization. For example, 74 percent of participants reported 
receiving help on how to look for a job only from the college, 21 percent reported they received such 
help from the college plus another organization, and 4 percent from only another organization. 
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Exhibit 4-14. Organization Providing Services 
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Source: Short-term participant follow-up survey 
Note: Among those receiving each service, N varies by row. 


| More than half of participants were highly satisfied with the grant-funded 
services they received. 


Nearly 60 percent were very satisfied with the training and services they received, and 33 percent 
reported being somewhat satisfied (not shown). A small minority (7 percent) were not satisfied. These 
figures include those still in training as of the survey. 
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4.4. OUTCOMES RELATED TO TRAINING PROGRESS 


The section presents outcomes related to participants’ training progress, including completing programs 
and continuing to additional training. 


| Most participants completed their training program by the time they 
responded to the survey, about 15 months after program entry. 


Fifty-nine (59) percent of participants reported finishing the classes for their grant-funded program, and 
51 percent also received an associated credential (Exhibit 4-15); thus, for the purposes of this report, 
51 percent of participants completed their program. Nearly 17 percent of participants reported that they 
had finished classes for the program, received an associated credential, and started an additional training 
program. 


Almost a quarter (24 percent) left their program without finishing the required classes. Seventeen (17) 
percent were still enrolled in their program at the time of the follow-up survey approximately 15 months 
after program entry. 


Exhibit 4-15. Training Outcomes at 15 months 
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Source: Short-term participant follow-up survey 

Note: Sample includes all survey respondents (VV = 2,211). Components may not sum to totals due to rounding. 
Median lag from program entry to interview was 14.8 months, with 90 percent of interviews occurring between 12 
and 18 months after program entry. 
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| More than half of participants planned to return to college. 


Fifty-two (52) percent of participants reported that they planned to return to college at some time in the 
future. Among all participants who planned to return, the average number of months until they expected 
to re-enroll was eight. 


| About one third of participants earned college credits. 


Thirty-two (32) percent of participants reported they earned college credits from participating in their 
program (not shown). Of those who earned credits, the average number of credits earned was 16. 


4.5. OUTCOMES RELATED TO EMPLOYMENT, EARNINGS, 
INCOME, AND RECEIPT OF PUBLIC ASSISTANCE 


This section presents participants’ outcomes related to employment, earnings, and income about 15 
months after program entry. It first discusses changes in their earnings and employment status over 
time. It then examines whether participants were working after exiting their program, and if so, the 
nature of their jobs (i.e., whether the job was closely related to the industry of their training program and 
whether it provided employee benefits). The chapter concludes by examining changes in study 
participants’ household income and receipt of public assistance benefits over the period between program 
entry and the short-term participant follow-up survey. 


Because this follow-up period is relatively short, caution should be used in interpreting the findings. It is 
likely that the follow-up period was too short to capture some credential awards and earnings as well as 
reductions in public assistance benefit receipt. In particular, given that 17 percent of the sample was still 
enrolled in their programs at the time of survey follow-up, not all outcomes are post-training outcomes. 

As noted below, participants still enrolled in training were excluded from some analyses. 
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| Rates of employment increased from the quarters just before program entry to 
five quarters after entry. 


According to NDNH data (which measures whether participants had any employment earnings during the 
quarter), participants’ employment rate over the three quarters prior to program entry was about 

71 percent, but then decreased in the quarter of program entry (QO) to 66 percent. Then starting in Q1, 
their employment rate increased, mostly continuing to do so for the next four quarters. As Exhibit 4-16 
shows, by the fifth quarter after program entry (Q5), 78 percent of participants were employed, an 
increase of 12 percentage points since QO and 7 percentage points relative to the period before program 
entry.*! Twenty-two percent were not employed in the fifth quarter (i.e., had no employment earnings). 
Although employment continued to increase throughout the follow-up period (through Q5), a majority of 
participants were employed during each quarter, and thus combined work and training. 


Exhibit 4-16. Average Employment by Quarter from Three Quarters Prior to the Quarter of 
Program Entry to Eight Quarters Later 
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Source: National Directory of New Hires 
Note: Sample includes all participants with valid SSN (WV = 2,355). 


41 The employment rates generated from the follow-up participant survey data are different from those produced from the 
NDNH data. This is because the two data sources have different samples. In the follow-up survey, only participants who 
were no longer enrolled in their program were asked whether they were currently employed (N=1,835). The NDNH 
sample includes all participants who had provided their SSN, regardless of whether they were still enrolled in training 
(N=2,355). 
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| Average quarterly earnings increased by $2,272 between Q(-3) and Q5. 
The research team compared participants’ average quarterly earnings three quarters prior to program 
entry to their average quarterly earnings five quarters after program entry.* NDNH data shows average 
earnings dropped in the quarter of program entry (QO) and then began to rise each quarter thereafter 
(Exhibit 4-17). By the fifth quarter following program entry, average quarterly earnings were $7,187, an 
increase of $2,272 (or 46 percent) from three quarters prior to program entry. 


Exhibit 4-17. Average Quarterly Earnings from Three Quarters Prior to the Quarter of Program 
Entry to Eight Quarters Later 


$8,000 - $7,187 
$7,000 - 


$6,000 - 


$5,000 


$4,000 


$3,000 - 
$2,000 - 


$1,000 - 


$0 T T T T T T T T T 


1 
Quarter 


Source: National Directory of New Hires 
Note: Sample includes all participants with valid SSN (WV = 2,355). Participants with zero earnings included. 


42 Unemployed persons are included in these analyses. Any quarter in which a participant had zero earnings is included. 
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| Participants who were employed in a job related to training had larger 
earnings increases than those employed in a job not related to training. 


Exhibit 4-18 shows that changes in quarterly earnings differs by employment status at follow-up. 
Participants who were employed in job related to their training at follow-up had higher earnings in the 
three quarters before enrollment than those employed in a job not related to training at follow-up. They 
also experienced a larger decrease in earnings in the quarter of program entry. By the fifth quarter after 
program entry, earnings for participants employed in a job related to their training at follow-up was 
$9,831, an increase of about $3,800 (63 percent) from three quarters prior to program entry. Earnings 
for participants employed in a job not related to their training grew to $8,080, an increase of about 
$2,900 (57 percent) from three quarters prior to program entry. 


Exhibit 4-18. Average Quarterly Earnings, by Employment Status 


$12,000 - 
$9,831 
$10,000 - $9,322 
$8,000 - 
$6,038 $6,100 $5,601 $8,080 
$6,000 - : $4,953 $9,289 $6,970 : 


5,326 
$4,000 | $5,142 $5 $5,156  ¢4.694 $4,638 


$2,000 - ——Employed in a job related to training 


—®—Employed in a job not related to training 


$0 T T T T T T T T T 


3 -2 -1 0 1 2 3 4 5 
Quarter 


Source: National Directory of New Hires 

Note: Sample includes participants who reported being employed in the short-term participant follow-up survey and 
who provided a valid SSN. N: 459 employed in a job related to training, and 793 employed in a job not related to 
training. 
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| Participants who finished their required classes had the highest average 
earnings five quarters after program entry. However, participants who left 
without finishing their classes experienced the largest earnings gains. 


Exhibit 4-19 shows average fifth quarter earnings and the change in earnings between three quarters 
before program entry to five quarters after program entry by enrollment status. In the fifth quarter after 
program entry, participants who had finished the required classes earned on average of $208 more than 
those who left without finishing and $1,243 more than those who were still enrolled in their program at 
the time of follow-up. Participants who left without finishing had the largest average increase in earnings, 
$2,715.% 


Exhibit 4-19. Change in Quarterly Earnings by Enrollment Status 


$8,000 $7,472 57.64 
$7,000 $6,229 
$6,000 
$5,000 
$4,000 
$2,715 
$3,000 $2,224 
$2,000 $1,422 
re sat 
$- 


Average Q5 earnings Change in earnings, Q -3 to Q5 


AStill enrolled in required classes 
@ Finished classes 
HiLeft without finishing 


Source: National Directory of New Hires 

Note: Sample includes participants who responded to the short-term participant follow-up survey and who provided a 
valid SSN. N: 298 still enrolled in required classes, 1,140 finished classes, and 431 left without finishing. Median lag 
from program entry to interview was 14.8 months, with 90 percent of interview occurring between 12 and 18 months 
after program entry. 


43 NDNH data indicate whether an individual is employed and his or her earnings during the quarter, but it does not include 
information on the number of weeks in which the individual worked, the number of hours worked, or hourly wage. 
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| Participants who finished their classes were nearly three times more likely to 
be employed in a job related to their training compared to participants who 
left without finishing 


Exhibit 4-20 presents the employment status of participants no longer enrolled in their training 
programs. Slightly more participants who finished classes were employed at the time of the short-term 
follow-up survey than were participants who left without finishing (82 percent versus 78 percent). 
However, a minority of each group were employed in an occupation that was closely related to their 
program. Still, participants who finished classes were several times more likely than those who left 
without finishing to report current employment in a position related to their program (40 percent versus 
14 percent). A slightly larger proportion of each group reported they were ever employed in such a job 
(44 percent of participants who finished their classes versus 16 percent of participants who left without 
finishing). 


Exhibit 4-20. Employment Status and Training-related Employment, by Enrollment Status 


90% 
82% 
80% 78% 
70% 
60% 
50% 44% 
40% 
40% 
30% 
20% 16% 14% 
0% 
Currently employed as of 12-month Ever employed after Currently employed after 
follow-up survey date finishing/leaving classes in finishing/leaving classes in 
occupation aligned with training occupation aligned with training 
program program 


A Finished classes @ Left without finishing 


Source: Short-term participant follow-up survey 

Note: Sample includes survey respondents who finished classes (/V = 1,316) or left without finishing 

(NW =519). Median lag from program entry to interview was 14.8 months, with 90 percent of interview occurring 
between 12 and 18 months after program entry. 
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| Most participants reported full-time employment once no longer enrolled in 
their program, and more participants who finished classes than those who left 
without finishing had jobs with benefits. 


Of the participants who were no longer enrolled in their training programs, more participants who 
finished classes than those who left without finishing reported full-time employment (71 percent versus 
61 percent) (Exhibit 4-21). The reverse was true of part-time jobs: 17 percent of participants who left 
without finishing classes and 11 percent of those who finished reported part-time employment. Moreover, 
larger percentages of participants who finished classes than those who left without finishing were 
underemployed: 12 percent of participants who finished classes and 8 percent of those who left without 
finishing were working part-time but wanted to work full-time. 


Exhibit 4-21. Employment Status at Survey Follow-up, by Enrollment Status 


80% 
71% 
70% 
61% 

60% 

50% 

40% 

30% 

20% 17% 

11% - 12% ‘ 
10% = 5 
0% — 
Employed full-time Employed part-time Employed part-time, want Employed part-time, do 
to work full-time (under- not want to work full-time 
employed) 


Afinished classes @Left without finishing 


Source: Short-term participant follow-up survey 

Note: Sample includes only survey respondents who finished classes (V = 1,316) or left without finishing 

(NV =519). It does not include those still in training. Median lag from program entry to interview was 14.8 months, 
with 90 percent of interview occurring between 12 and 18 months after program entry. 
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As shown in Exhibit 4-22, more participants who finished classes than those who left without finishing 
reported employment in jobs that offered employee benefits. Sixty-one percent of participants who 
finished classes were in jobs that offered health insurance, compared to 51 percent of participants who 
left without finishing. The majority of participants who finished classes also reported their jobs offered 
paid vacation, paid holidays, and retirement or pension benefits, compared to 46 percent or fewer 
participants who left without finishing, depending on the benefit. 


Exhibit 4-22. Employed in Job with Benefits, by Enrollment Status 


70% 
61% — 
60% ° a8% 
54% 
51% 49% 
50% 46% 46% 
2 45% 43% 
40% 38% 
34% 

30% 
20% 
10% 

0% 

Health insurance Paid vacation Paid holidays Retirement or Paid sick days Health insurance 
pension benefits and paid sick 
days 


AfFinished classes @Left without finishing 


Source: Short-term participant follow-up survey 

Note: Sample includes survey respondents who finished classes (/V = 1,316) or left without finishing 

(NW =519). Median lag from program entry to interview was 14.8 months, with 90 percent of interview occurring 
between 12 and 18 months after program entry. 


| Over the 15-month follow-up period, participants’ household income 
increased and the percentage living in poverty decreased, but public 
assistance benefits receipt did not change. 


Average household income increased from $33,944 at the time of program entry to $38,148, according to 
the short-term participant follow-up survey, an increase of $4,204, or 12 percent (not shown). As 
Exhibit 4-23 shows, the proportion of study participant households in poverty decreased during the 
follow-up period, from 42 percent at program entry to 34 percent 15 months after entry, a decrease of 

8 percentage points. 


Exhibit 4-23 also shows public assistance benefit receipt. The proportion of study participants receiving 
SNAP, TANF, or TRA remained consistent between program entry and the end of the follow-up period, 
with very few participants receiving benefits. 
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Nineteen (19) percent of participants reported receiving means-tested public assistance (including TANF 
and SNAP) as well as other federally funded assistance programs (e.g., TRA and Social Security Disability 
Insurance (SSDI)) at about 15 months after program entry (not shown). Change over time is not 
presented in Exhibit 4-23 for this broader measure of public assistance benefits receipt because of 
wording changes in the survey from program entry to follow-up.“ 


Exhibit 4-23. Household Poverty Status and Receipt of Public Assistance Benefits over Study 


Period 
45% 42% 
40% 
35% 34% 
30% 
25% 
20% 
15% 
10% 
5% 
0% 


Poverty 


11% 11% 


9% 8% 
A ® 
2% 1% A 
= a 


TANF SNAP Poverty and 


receiving either 


TANF or SNAP 


AProgram Entry @Follow-up 


Source: Short-term participant follow-up survey 
Note: Sample includes all survey respondents (N = 2,211). Median lag from program entry to interview was 14.8 
months, with 90 percent of interview occurring between 12 and 18 months after program entry. 


A ® 
2% 2% 

a) 
TRA 


44 At program entry, participants reported receipt of TRA, TANF, SNAP, and Unemployment Insurance on the Baseline 
Information Form. About 15 months later, they reported receipt of TRA, TANF, SNAP, and “any other federally funded 
programs such as Social Security Disability Insurance (SSDI) or Supplemental Security Income (SSI), veterans’ benefits, 
Housing Choice Voucher Program (formerly known as Section 8), or Public Housing” on the short-term participant follow- 


up survey. 
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Poverty status and public assistance receipt at follow-up varied by enrollment status. As shown in 
Exhibit 4-24, 32 percent of those who finished their classes were below the poverty level, compared to 
37 percent of those who left without finishing and 39 percent of those still enrolled. SNAP receipt was 
highest for those who left without finishing (15 percent, compared to 10 percent for those who finished 
their classes or were still enrolled). 


Exhibit 4-24. Household Poverty Status and Receipt of Public Assistance Benefits at Follow-up, 


by Enrollment Status 


45% 
40% 37%) 92% 
35% 32% 
30% 
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TANF SNAP Poverty and 


receiving either 
TANF or SNAP 


@Left without finishing wm Still enrolled 


Source: Short-term participant follow-up survey 
Note: Sample includes all survey respondents (WV = 2,211). Median lag from program entry to interview was 14.8 
months, with 90 percent of interview occurring between 12 and 18 months after program entry. 
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5. Participant Outcomes, by 
Characteristics at Program Entry 


This chapter explores whether outcomes varied by participants’ socioeconomic, demographic, and 
educational characteristics at program entry. In doing so, it addresses the following research question: 


The subgroup analysis provides important insights about who benefits most (and least) from programs 
such as those funded by TAACCCT under the general economic conditions prevailing during the study 
period, between late 2017 and early 2019. Such an understanding can help policymakers and program 
designers refine program eligibility requirements or implement future programs that achieve more 
equitable outcomes across socioeconomic, demographic, and educational groups. 


Different outcomes across subgroups may be due to the subgroup status itself or due to other factors 
that are frequently associated with that subgroup. For example, in the analysis presented below, a larger 
percentage of participants who had some postsecondary education at program entry completed their 
program than participants without such education. This difference may be due to the prior education 
itself or to factors commonly associated with those who have more education, such as more motivation 
and higher socioeconomic status. The analysis does not differentiate between these possibilities. 


Given the large number of characteristics measured at program entry and outcomes measured 15 months 
later, the research team focused on 10 characteristics at program entry and four outcomes. These 
choices were guided by the TAACCCT theory of change presented in Chapter 2. 


Key Takeaways: Participant Outcomes by 
Characteristics at Program Entry 


Older participants and those with some postsecondary education had higher rates of 
'e)coye) r-}anmee)an}®)(=14(0)0 


Training-related employment was lower for public assistance recipients, those with less 
education, and those who had not worked in the year prior to enrollment. 


Earnings increases were larger for younger participants and lower for those who were not 
employed at enrollment but worked in the previous year. 


Participants enrolled in longer programs experienced larger earnings increases than those 
enrolled in shorter training programs. There was no detectable difference in program 
completion, training-related employment, or public assistance receipt by program duration. 


Participants previously employed in manufacturing had higher rates of training-related 
employment and larger earnings increases than those employed in certain other industries. 


Older participants, those with long periods without employment, and those receiving public 
assistance at program enrollment had higher rates of public assistance receipt a year later. 
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Sex, 

Race/ethnicity, 

Veteran status, 

TRA receipt, 

TANF and SNAP receipt, 

Prior postsecondary education, 
Age, 

Time since last job, 

Previous industry, and 


Program duration.*° 


The follow-up outcomes are:*° 


Program completion, 
Training-related employment, 
Change in earnings, and 


Public assistance benefit receipt. 


Interpreting Exhibits in 
Chapters 5 and 6 


The purple circles in each exhibit are the 
observed average outcomes for the particular 
subgroup. The blue lines extending sideways 
from each circle indicate the length of a 

95 percent confidence interval. 


The research team conducted statistical tests to 
determine which differences between subgroups 
were statistically significant. 


e For subgroups with two categories (e.g., Sex: 
male, female), a single test was conducted to 
determine whether the difference was 
significant. 


For subgroups with more than two categories 
(e.g., race/ethnicity: black (non-Hispanic), 
white (non-Hispanic), Hispanic, etc.), the 
category with the most members was 
designated as the “reference group” (labeled 
with “[reference group]” in the exhibits). 
Statistical tests for differences were 
conducted between the reference group and 
each of the other categories in that subgroup. 


In the exhibits, differences that are significant at 


the .05 level are identified with dark purple 
circles; non-significant differences are identified 
with light purple circles. 


Many of the exhibits in this and subsequent 
chapters show estimates with confidence 
intervals. A confidence interval is a measure of 
uncertainty. Though there is very little 
uncertainty about outcomes for the participants who enrolled in the 34 programs during the study period, 
there is considerable uncertainty about outcomes for future participants—especially what outcomes might 
be for participants who enroll in programs offered by new grantees if there were another round of 
TAACCCT grants or similar types of grant programs. The confidence intervals in this report represent the 
range of possible outcome levels that would be expected for programs from a new round of grants, 
assuming the same general economic conditions and same grant competition rules. Clearly these 
conditions would not apply during the COVID-19 pandemic, but they could return in a few years, near the 
end of the next expansionary business cycle. More information on the calculation and interpretation of 
these confidence intervals is provided in Appendix B, Section B.5. 


Outcomes levels are presented for each subgroup, along with a 95 percent confidence interval. 
Confidence intervals are narrower (reflecting less uncertainty) for outcomes based on larger sample sizes 


45 See Appendix C for detailed descriptions of the measurement strategies for these subgroups. The research team studied 
subgroups defined by combinations of age and time since last job. 


46 See Section 2.3 of Chapter 2 for additional discussion of why the team chose these variables for closer examination 
across Chapters 5, 6, and 7. 
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and for outcomes that are more consistent across programs. Confidence intervals are wider (reflecting 
more uncertainty) for outcomes based on smaller sample sizes and outcomes that vary widely across the 
34 programs. For example, outcomes based on NDNH data generally have smaller confidence intervals, 
because almost all study participants are included in the sample. Confidence intervals are generally larger 
for subgroups, especially those with small sample sizes (e.g., TANF recipients). 


The research team conducted statistical tests to determine which differences between subgroups were 


statistically significant.*” In cases where there is more 
than one group within a category, the largest group was 
designated as the “reference group” and statistical tests 
for differences were conducted between the reference 
group and each of the remaining categories.*® The 
statistical tests in this chapter account for variation in 
outcome levels across programs. In general, differences 
will only be significant if the differences are fairly 
consistent across the 34 programs. 


The remainder of this chapter is organized by the four 
outcomes. Detailed tables underlying the figures are 
available in Appendix G. 


5.1. PROGRAM COMPLETION 


This section explores how the share of participants who 
completed their program varied by subgroup. As noted 
in Chapter 4, 51 percent of all participants “completed” 


Survey Questions: 
Program Completion 


From the short-term participant 
follow-up survey: 


Are you still taking classes required for 
the [PROGRAM NAME] at [SCHOOL 
NAME], or did you complete the 
required classes, or did you stop 
without completing them? If you are 
between terms, please consider that as 
currently taking these classes. 


e Still taking classes 


e Finished the classes 


e Stopped without finishing the classes 


[IF NOT STILL TAKING CLASSES] Were 
you awarded one or more industry- 


recognized certificates, licenses, or 
credentials as a result of completing 
the required classes for the [PROGRAM 
name] at [SCHOOL NAME]? 


their program of study; that is, they finished classes for 
their program and received a credential by about 15 
months after program entry. 


Exhibit 5-1 shows program completion, by participant e Yes 
characteristics at program entry. As detailed below, Pa \(0) 
older participants and those with some postsecondary 
education had higher rates of program completion than did younger participants with no postsecondary 
education. 


| There was no detectable difference in program completion by sex, veteran 
status, public assistance benefit receipt, or program duration. 


About 50 percent of men completed their program, with a confidence interval that ranges between 
39 percent and 61 percent. About 59 percent of women completed their program, with a confidence 


4” Tests were simple two-sided two-sample tests based on standard errors calculated as described in Appendix B, 
Section B.5 and the assumption of asymptotic normality. If the absolute difference between the mean outcomes across 
the two groups was more than 1.96 times as large as the estimated standard error on the difference, then the pair of 
domains was flagged as having statistically different mean outcomes. 


48 Alternative procedures would have been to test for at least one difference or to test all possible pairwise differences. The 
research team found it most interesting to always compare less common groups to the most common group. 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1] 56 


CHAPTER 5: PARTICIPANT OUTCOMES, BY 
CHARACTERISTICS AT PROGRAM ENTRY 


interval that ranges between 45 percent and 73 percent. The difference in program completion between 
men and women was not significant, nor was the difference between veterans (44 percent) and non- 
veterans (52 percent). Similarly, there were no detectable differences in program completion for 
subgroups defined by receipt of public assistance benefits (TRA, TANF, or SNAP) or by program duration. 


| There was no detectable difference in program completion for any 
race/ethnic group as compared to white (non-Hispanic). 


For race and ethnicity, white (non-Hispanic) is the largest subgroup, designated as the reference group. 
None of the differences between white (non-Hispanic) and the other three groups were statistically 
significant. 


| Participants age 25 and older had higher program completion rates than 
younger participants, regardless of prior employment. 


Exhibit 5-1 also examines program completion by age and time since last job. This analysis includes 
nine categories; the largest category is age 24 or younger and currently employed, designated as the 
reference group. Participants age 25 to 34 had significantly higher program completion rates regardless 
of their employment history than had the reference group, as did participants age 35 or older with past 
or current work experience. The analysis does not provide insight into the cause of these differences; it is 
nonetheless interesting that most of the older groups are more successful than young participants at 
completing their program. Within each of the age groups, those who were not working at program entry 
but who had some employment in the past year had the highest rates of program completion—perhaps 
because those workers were able to participate in their program full-time. 


| Participants with some postsecondary education at program entry had higher 
program completion rates than those without any postsecondary education. 


Fifty-five (55) percent of those with some postsecondary education at program entry completed their 
program, compared to 47 percent of those with no postsecondary education; this difference is statistically 
significant, as noted with the dark purple circles in Exhibit 5-1. A possible conclusion from this finding is 
that prior postsecondary experience helps participants to finish their courses and earn the associated 
credential. However, as noted earlier, prior postsecondary experience could merely be associated with 
other characteristics (e.g., cognitive ability, motivation, social and family supports) that are producing the 
different outcomes. 


| Compared to manufacturing, the most common former industry, there was no 
compelling evidence that study participants with former experience in other 
industries had different program completion rates. 


Exhibit 5-1 also shows program completion by the industry in which participants worked before 
enrolling in their grant-funded program, for those not employed at program entry. The analysis includes 
eight categories; manufacturing is designated as the reference group. Although there is variation in the 
estimated program completion rates across industries, there was no detectable difference in program 
completion between manufacturing and any of the other seven industries. 
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Exhibit 5-1. Program Completion, by Participant Characteristics at Program Entry 
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Other 
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Non-veteran 
Age 24 or less, 


employed 
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Characteristics at Program Entry 


ap 24 or less, 
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Age 24 or less, 
no work in past year 
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employed 


Age 25 to 34, worked 
in past year 


Age 25 to 34, 

no work in past year 
Age 35 or older, 
employed 


Age 35 or older, 
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Age 35 or older, 
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KEY @ Statistically significant difference (p < 0.05) 


C ) Statistically insignificant difference 


Source: Baseline Information Form and short-term participant follow-up survey 

Note: Sample size is 2,211 survey respondents; for former industry, sample is limited to participants who were not 
employed at program entry but had some previous work experience (N=883). Average is denoted with circles; 
bounds of 95 percent confidence interval are denoted with squares. For subgroups with more than two categories, 
the category with the largest number of members is designated as the “reference group,” and tests for differences 
were conducted between the reference group and each of the other categories. Expanded results are reported in 


Appendix G. 
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5.2. EMPLOYMENT AND EARNINGS OUTCOMES 


This section explores how select labor market outcomes—training-related employment and change in 
earnings—varied across subgroups. The team chose to focus on these two labor market outcomes 
because together they cover both current and future benefits of training. Even if study participants do not 
immediately start earning high wages, the fact that they have 


training-related employment suggests potential that their Study 
earnings will eventually increase as a result of the training. If Population 
on the other hand, they cannot obtain employment relevant The sample in this section 


to their training, it seems very unlikely that they will ever includes only participants who had 
finished their classes or left 


without finishing at the time of 
survey follow-up. 


benefit financially from the training, 


As discussed in Chapter 4, participants who finished their 
classes had more success obtaining training-related This decision was mandatory for 
employment than participants who left without finishing. training-related employment 
Because of this, the team considered repeating subgroup pieced ee pees 
analyses separately for participants who finished their classes impose on analysis of NDNH data. 
versus those who left without finishing, but ultimately decided 
against such an analysis. There were several reasons for this decision. First and most importantly, no 
participant enters a program with the intent to stop without finishing it, so such analyses would not be 
relevant to any specific population. Second, it would have doubled the number of comparisons, making it 
difficult for readers to take away a clear message. Third, participants who finished their classes were not 
more successful in the other primary outcome of this section—earnings growth. Fourth, there were 
anecdotal reports that the decision to stop attending classes before finishing was not necessarily a sign of 
failure. Discussions during site visits indicated that for some programs, employers were so eager for 
employees that employers recruited program participants before they finished their grant-funded training. 


Another important caveat is that the follow-up period is relatively short for assessing employment and 
earnings outcomes, particularly for longer programs. The outcomes presented in this section may change 
over time as the longer-term effects of grant-funded training are fully realized. For example, the 
evaluation of the Project QUEST program found that it did not produce impacts on earnings until several 
years after program entry.*? 


49 The earnings impacts for Project QUEST’s participants did not emerge until the fourth year and continued for nine years 
(Roder and Elliot 2019). Recognizing that it can take time for impacts to emerge, studies of job training programs are 
increasingly using longer follow-up periods. For example, the WIA Gold Standard evaluation had a three-year follow-up 
period (Fortson et al. 2017). Some evaluations have included much longer follow-up periods, such as the National Job 
Corps study, which collected earnings data for more than 10 years (Schochet et al. 2006). In addition, the evaluations of 
the Health Profession Opportunity Grants (HPOG) and Pathways for Advancing Careers and Education (PACE) programs 
are using a six-year follow-up period (https://www.acf.hhs.qov/opre/research/project/career-pathways-long-term- 
outcomes-study). 
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5.2.1 Training-Related Employment 
via 7 Survey Questions: Training- 
Exhibit 5-2 shows the share of participants employed 


ae aie: ae Related Employment 
in a job related to their training, by participant 
characteristics at program entry. As reported in From the short-term participant 
Chapter 4, across all participants, 33 percent were follow-up survey: 
employed in a job closely related to their training after How closely related ig this job to the 
finishing or leaving training. As discussed below, education and training you received as 
training-related employment varied across a number part of [PROGRAM NAME] and [SCHOOL 
of subgroups. NAME]? Would you say... 

| Black (non-Hispanic) participants had e Closely related 


lower rates of training-related 
employment than white (non- 


e Somewhat related 


Hispanic) participants. e Not related 
Exhibit 5-2 also shows that black (non-Hispanic) If the respondent was currently i 
participants had significantly less success obtaining a employed and answered closely related 
: . ae ; : to this question, then the research team 
job related to their training than did white (non- classified that participant has having 
Hispanic) participants (22 versus 36 percent). There training-related employment. 


were no significant differences in training-related 
employment by sex or veteran status. 


| SNAP recipients and participants with no postsecondary education had lower 
rates of training-related employment. 


Exhibit 5-2 also shows that SNAP recipients (22 percent) had significantly less success obtaining job- 
related training than non-recipients (34 percent). Note that this disparity is not due to differences in 
enrollment status because, as discussed above, SNAP recipients finished their training courses and 
earned a credential at a rate not significantly lower than participants not receiving SNAP. In addition, 
participants with no postsecondary education at program entry (28 percent) had significantly lower rates 
of employment in a job related to training than those with some postsecondary education (37 percent). 
This disparity could be due to differences in program completion rates because, as discussed above, 
participants without prior postsecondary experience were much less likely to complete their program. 
There were no detectable differences in training-related employment by TRA or TANF receipt at program 
entry, nor by duration of program. 


| Participants age 25 to 34 and those with more recent employment had the 
highest rates of training-related employment. 


Exhibit 5-2 also shows that rates of training-related employment differed across subgroups defined by 
age and prior employment. Participants of ages 25 to 34 who were employed at program entry had a 
significantly higher rate of training-related employment (41 percent) than those ages 24 or younger who 
were employed at program entry (33 percent). 


Participants with more recent employment also had higher rates of training-related employment. 
Participants age 24 or younger employed at program entry or who had worked in the past year had 
significantly higher rates of training-related employment than participants of the same age who were not 
employed in the year prior to program entry. 
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Exhibit 5-2. Training-related Employment, by Participant Characteristics at Program Entry 


Percent (%) Percent (%) 
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Source: Baseline Information Form and short-term participant follow-up survey 

Note: Sample size is 1,834 survey respondents who had finished their classes or left without finishing by about 15 
months after program entry; for industry, sample is limited to participants who were not employed at program entry 
but had some previous work experience (N=756). Average is denoted with circles; bounds of 95 percent confidence 
interval are denoted with squares. For subgroups with more than two categories, the category with the largest 
number of members is designated as the “reference group,” and tests for differences were conducted between the 
reference group and each of the other categories. See Appendix G for expanded results. 
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| Participants previously employed in manufacturing had higher rates of 
training-related employment than those employed in other industries. 


Exhibit 5-2 also shows there were differences in training-related employment by prior industry. Forty 
(40) percent of participants who previously worked in the manufacturing sector were employed in a job 
related to their training. This was a higher rate than participants who were previously employed in trade, 
transportation, and utilities (28 percent), professional and business services (25 percent), or leisure and 
hospitality (20 percent). 


5.2.2 Change in Earnings 


Among participants who had finished their classes or left without finishing, average quarterly earnings 
increased by $2,359 between the third quarter before program entry and the fifth quarter after program 
entry (or about $9,500 per year). As shown in Exhibit 5-3, and discussed in more detail below, the 
change in earnings varied across a number of subgroups. 


| There were no statistically significant differences in the change in earnings by 
sex or veteran status. 


Exhibit 5-3 shows that there were no differences in training-related employment or quarterly earnings 
by sex or veteran status. Quarterly earnings data were not available for race/ethnicity subgroups, so they 
are not included in this analysis. *° 


| TRA recipients had a decrease in quarterly earnings. 


Exhibit 5-3 also shows that TRA recipients had a decrease in quarterly earnings of $2,218, whereas 
non-recipients saw an increase of $2,443. This finding is possibly due to TRA recipients being workers 
laid off from typically higher-paying trade-affected industries, such as manufacturing. TRA recipients had 
earned an average of nearly $8,500 in the third quarter before program entry, substantially more than 
non-TRA recipients, who had earned about $5,000 in that same quarter. TRA participants likely had 
challenges obtaining a similarly high-paying job in the year after program entry. 


| SNAP recipients had smaller earnings increases than non-recipients. 


Exhibit 5-3 also shows that SNAP recipients experienced an increase in quarterly earnings of $1,511, 
compared to $2,471 for non-recipients. There was no detectable difference in earnings change by TANF 
receipt at program entry. 


| Participants enrolled in longer programs had significantly larger earnings 
increases than those enrolled in shorter programs. 


Exhibit 5-3 also shows that participants enrolled in programs longer than one semester had more than 
double the increase in quarterly earnings ($3,358) as those enrolled in shorter programs ($1,680). It is 
possible that longer programs were more beneficial to participants and produced larger earnings 
increases than shorter programs. However, it is also possible that participants who enrolled in longer 
programs were different in other ways from participants who enrolled in shorter programs, and that those 
differences instead led to the larger increase in earnings. 


50 Because of an oversight, ethnicity data was not available on the file that could be merged with earnings data. 
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Exhibit 5-3. Two-Year Change in Quarterly Earnings, by Participant Characteristics at Program 
Entry 
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Change in Quarterly Earnings ($) 
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[reference group] 
Natural Resources 
and Mining 3920 i 
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Source: National Directory of New Hires and Baseline Information Form 

Note: Sample size is 1,571 survey respondents who had finished classes or left without finishing by the short-term 
follow-up survey and had sufficient information at program entry for matching to NDNH wage records; for former 
industry, sample is limited to participants who were not employed at program entry but had some previous work 
experience (N=688). Quarterly earnings data were not available for race/ethnicity subgroups. Average is denoted 
with circles; bounds of 95 percent confidence interval are denoted with squares. For subgroups with more than two 
categories, the category with the largest number of members is designated as the “reference group,” and tests for 
differences were conducted between the reference group and each of the other categories. Expanded results are 
reported in Appendix G. 


| Younger participants—especially those employed at program entry or who 
had not worked in the past year—experienced the largest earnings gains. 


Exhibit 5-3 also shows that in contrast to the results for program completion and training-related 
employment, earnings increases were largest for younger participants, especially those employed at 
program entry. Quarterly earnings rose by $4,061 for participants age 24 or less who were employed at 
program entry, compared to an increase of $2,159 for those age 25 to 34 and $2,494 for those age 35 or 
older. One possible explanation for this finding is that younger workers had lower earnings at program 
entry than older workers, and due to their age came to the program with less professional knowledge 
and experience. If this were the case, the program would have provided a larger relative increase in skills 
for younger workers than older workers, which in turn would lead to larger earnings gains. Across age 
categories, those who were not employed at program entry but who worked in the previous year had the 
smallest earnings gains. 


| Participants previously employed in professional and business services 
experienced a net earnings loss over the two-year period. 


Exhibit 5-3 also shows that those who were previously employed in manufacturing experienced a larger 
earnings gain than those previously employed in professional and business services. Participants 
previously employed in manufacturing had an increase in quarterly earnings of $2,052, whereas those 
employed in professional and business services had a $266 decrease. 
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5.3. RECEIPT OF PUBLIC ASSISTANCE BENEFITS 


As reported in Chapter 4, nearly one-fifth (19 percent) of participants reported receiving means-tested 
public assistance benefits (including TANF and SNAP) as well as other assistance from other federally 
funded programs (e.g., TRA and SSDI) by about 15 months after program entry. This section explores 
whether this rate varied across subgroups. 


As shown in Exhibit 5-4 and discussed in more detail below, black (non-Hispanic) participants, those 
who were receiving benefits at program entry, older participants, and those with long periods 
unemployed had higher rates of public assistance benefit receipt. 


Public Assistance Programs included in this Study 


TANF (Temporary Assistance for Needy Families) 


SNAP (Supplemental Nutrition Assistance Program) (formerly called the Food Stamps 
Program) 


TRA (Trade Readjustment Allowances 


Other federally funded programs such as Social Security Disability Insurance (SSDI) or 
Supplemental Security Income (SSI), veterans’ benefits, Housing Choice Voucher Program 
(formerly known as Section 8), or Public Housing 


| More black (non-Hispanic) than white (non-Hispanic) participants received 
public assistance benefits. 


Twenty-seven (27) percent of black (non-Hispanic) participants received public assistance by about 15 
months after program entry, compared to 16 percent of white (non-Hispanic) participants. 


| More veterans received public assistance benefits. 


Among veterans, 31 percent received benefits, much higher than the rate among non-veterans 

(18 percent). One factor that likely affected this result is that one of the benefits included in the outcome 
was veterans’ benefits, which only veterans are eligible to receive. The study did not examine variation in 
receipt of just means-tested public assistance benefits by veteran status. 


| Most participants who received SNAP, TANF, or TRA benefits at program entry 
were still receiving public assistance benefits 15 months later. 


As discussed in Chapter 4, overall receipt of public assistance benefits barely changed from program 
entry to 15 months later, but this was due to a mixture of people stopping benefits and others starting 
them. Unsurprisingly participants who received SNAP, TANF, or TRA benefits at program entry received 
benefits at significantly higher rates at follow-up than those who were not receiving these benefits at 
program entry,. For example, 65 percent of participants who received TANF at program entry were 
receiving some type of public assistance benefit at follow-up, compared to 19 percent of participants who 
did not receive TANF at program entry. There were no differences in public assistance benefit receipt for 
subgroups defined by educational attainment or program duration. 
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Exhibit 5-4. Public Assistance Benefit Receipt 15 Months after Program Entry, by Participant 
Characteristics at Program Entry, Including Receipt of Public Assistance from Selected 
Federal Programs 
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Source: Baseline Information Form and short-term participant follow-up survey 

Note: Sample size is 2,211 survey respondents; for former industry, sample is limited to participants who were not 
employed at program entry but had some previous work experience (N=883). Average is denoted with circles; 
bounds of 95 percent confidence interval are denoted with squares. For subgroups with more than two categories, 
the category with the largest number of members is designated as the “reference group,” and tests for differences 
were conducted between the reference group and each of the other categories. Expanded results are reported in 
Appendix G. 
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| Eleven percent of younger participants employed at program entry received 
public assistance benefits, a rate lower than most other groups defined by age 
and duration of unemployment at program entry. 


Public assistance benefit receipt differed by age and length of unemployment at program entry. Crossing 
three levels of age (24 years or less, 25 to 34 years, and 35 years or more) by three levels of length of 
unemployment (employed, employed within the prior year, and not employed within the prior year) yields 
nine subgroups. Seven of these subgroups had significantly higher rates of benefit receipt than the 
reference category (24 years or less and employed). For example, 40 percent of those age 35 or older 
who had not worked in the year prior to program entry were receiving public assistance benefits, 
compared with 11 percent of those age 24 or less and employed. 


| Among those with some employment within the year prior to program entry 
but not employed at that time, public assistance benefit receipt varied by 
industry of prior employment. 


As noted previously, manufacturing was the most common prior industry for participants who were not 
employed at program entry but who had been employed within the prior year. Compared to this 
reference group, one prior industry had a rate of public benefits that was significantly lower. Despite their 
net loss in earnings ($266 decrease in earnings, as reported in Exhibit 5-3), among participants who 
last worked in professional and business services, only 21 percent received public assistance benefits at 
follow-up. This is significantly lower than participants who last worked in manufacturing (32 percent). 
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6. Participant Outcomes, by Service 
Receipt 


Chapter 5 presented simple differences in average outcomes between 


Warning 
subgroups defined by participant characteristics at program entry without 
any adjustment for other differences between the subgroups. This chapter The analysis methods 
takes a different approach. It presents simple differences in average in this chapter make 


strong assumptions. 


outcomes between subgroups defined by the receipt of supportive i 

. : : eee . Please be advised 
services and modes of instruction, and it adjusts these differences for that results are not as 
known differences at program entry between the subgroups. The hope is definitive as those in 
that these adjusted differences will approximate the impacts of the prior chapters. 
services and modes of instruction, but caution is required for this reading, 
as is explained below. Full methodological details may be found in Appendix D. See the text box on the 
next page for a condensed and less technical description of the methods employed in this chapter. 


This chapter addresses the following research question: °! 


Do study participants who utilize Key Takeaways: Service Receipt 


various services have better and Participant Outcomes 
outcomes, after controlling for 


characteristics at program entry? 


The combination of work-based learning and 


This is an important question because, as employment-related services is strongly associated 
reported in Chapter 4, most participants with program completion and training-related 

did not have postsecondary education employment. 

experience when they entered their Skills practice in work-like physical environments is 
program and many were older, meaning associated with program completion and stronger 
that many years had passed since their last earnings growth. 

time in school. Returning to school can be Transfer credits, financial aid advising, all three 

a daunting challenge. TAACCCT programs work-based learning services, and both employment 


related services had positive associations with 


implemented a range of supportive training-related employment. 


services beyond basic classroom 
instruction, with the aim of helping Transfer credits had a positive association with 
earnings growth among participants not still 


participants persist in and complete their enrolled in the program as of follow-up. 


programs. However, it is difficult to 
determine the effect of services on Three of the four persistence and completion 
services were associated with public assistance 


petenton and pomneoon: Some supportive receipt, while none had significant associations with 
services may be ineffective or even poverty. 


counterproductive. So estimates of service 
efficacy should be useful to designers of 
future college-based programs for adult 
learners. 


Opportunities for direct occupational experience 
were also associated with public assistance receipt. 


51 Full methodological details may be found in Appendix D. See the text box on the next page for a condensed and less 
technical description of the methods employed in this chapter. 


Abt Associates Round 4 Early Outcomes Study Report December 2020 ] 68 


CHAPTER 6: PARTICIPANT OUTCOMES, 
BY SERVICE RECEIPT 


Analytic Methods and 
Interpreting Adjusted Differences 


Please see Appendix D for full details on methods employed in this chapter. In brief, this 
chapter estimates adjusted differences in average participant outcomes across subgroups 
defined by receipt of supportive services and modes of instruction. The adjusted differences 
are based on comparing predicted and actual outcome levels for participants who receive a 
particular service and those who do not. The basic analytic method is as follows: 


For each service and each outcome: 


1. Predict the outcome level for those participants who received the service and those who 
did not, using participant characteristics at program entry. This is the expected outcome 
rate. 


. Calculate the actual level of the outcome for those who received the service and those 
who did not. This is the actual outcome rate. 


. Calculate the difference between the actual and expected outcome rates, separately for 
those who received the service and those who did not. This is the excess outcome rate. 


. Calculate the difference between the excess outcome rate for those who received the 
service and those who did not. This difference is the adjusted difference between those 
receiving and those not receiving the service. Under strong assumptions explained below 
(and in more detail in Appendix D), this adjusted difference is a consistent estimate of the 
impact of the service. 


The research team performed tests for statistical significance of the adjusted differences. 
Adjusted differences with a p-value of less than .05 are considered to be statistically 
significant. This means that if the true adjusted difference were zero, then the probability of 
observing an estimated adjusted difference as large or larger would be less than 5 percent. 


Because service receipt was not randomly assigned, interpretation of the adjusted differences 
as impacts requires that none of the following conditions are true: 


e The decision to utilize a service is related to an unmeasured factor at program entry that 
also directly influences the likelihood of achieving the target outcome; 


The decision to utilize a service is related to early academic progress that is different than 
what would be expected based on factors measured at program entry; 


e Local availability of the service is related to unmeasured local labor market conditions; or 


e The service is bundled with other services. 


While it seems implausible that none of these conditions are ever true, the adjusted 
differences almost certainly provide better clues and hints about the efficacy of the various 
services that an one could obtain from the unadjusted difference because the research team 
was able to collect a rich set of characteristics at program entry to use in the adjustment. 
The text accompanying many of the statistically significant estimated impacts explores 
possible reasons other than service receipt for the adjusted differences. 
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The answer to this question is necessarily more tentative than the answers to the questions of the prior 
chapters. The basic research technique is to compare outcomes for those who received a service with 
outcomes for those who did not receive the service. However, there can be many reasons why some 
people utilize a service when others do not. Service receipt may be associated with a range of other 
factors that affect participant outcomes but have little to do with the service itself. 


To approximate the impact of each service, the research team prepared adjusted differences. Strong 
assumptions about the reasons for service receipt and the potential pathways from program entry 
through service receipt to training outcomes are necessary for the adjusted differences to be safely 
interpreted as service impacts. The box Analytic Methods and Interpreting Adjusted Differences 
above gives an overview of the team’s analytic approach and examples of ways that these strong 
assumptions might fail to be true. Readers should be mindful that the adjusted differences estimated in 
this chapter are not as credible as they would have been had it been possible to randomly assign services 
to participants. Some of the adjusted differences 


are more plausible estimates of impacts than Measurement of 

others; the detailed text in this chapter discusses Service Receipt 

the plausibility of each result. As mentioned 

earlier, for extensive detail on the methodology, Accelerated and Enhanced Learning 

see Appendix D. 1. Transfer credits (earned previously at the 
same college or at other colleges), or 

Given the variety of services and outcomes 2. Credit for prior learning (e.g., in the 

measured, the potential number of connections military, apprenticeship programs, or 

that could be estimated is very large. To make this other work experiences). 

exploration more digestible, the team chose to Persistence and Completion 

focus on the connections of 11 services with four 3. Career counseling, 

outcomes. The selected 11 services are 4. Academic advising, 

prominently featured in grantees’ plans. The list is 5. Financial aid advising, or 

not meant to be exhaustive and very well may 6. Workplace skills/general life skills. 

exclude important services (the availability of Connections to Employment—Work-based 

instructors for after-class consultation, for Learning 

example). The data source—the short-term 7. Skills practice in a virtual setting, 

participant follow-up survey—included questions 8. Skills practice in a work-like physical 


environment, or 
9. In-program work experience 
(e.g., internship). 


about service receipt. Those questions aligned 
with the three categories of capacity-building 
strategies described in the TAACCCT theory of 
change (Chapter 2), although the wording differs 
slightly and the connections to employment 
category is divided into two subcategories ( work- 
based learning and employment-related services): 


Connections to Employment—Employment- 
Related Services 


10. Job search or placement assistance, or 
11. Interviewing practice. 


As in Chapter 5, these analyses focus on four outcomes: 


e Program completion; 


e  Training-related employment; 
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e Change in earnings; and 


e Receipt of public assistance benefits>* 


The rest of this chapter is organized by these four outcomes. Within each section, the impacts (adjusted 
differences) of the 11 services on the focal outcome are presented and discussed. The discussion of 
impacts of the services on program completion in Section 6.1 is more detailed than the corresponding 
discussions in the remaining sections—specifically, Section 6.1 provides more detail on the methodology. 
Detailed tables underlying the figures presented in this chapter are available in Appendix G. 


6.1. PROGRAM COMPLETION, BY SERVICE RECEIPT 


This section focuses on connections between service receipt and program completion.°? Exhibit 6-1 
compares the expected completion rate to the actua/completion rate for study participants who received 
a particular service and those who did not. The adjusted difference of each service on program 
completion is equal to the difference between actual completion rate and expected completion rate for 
participants who received each service, minus the difference between actual and expected completion 
rates for participants who did not receive each service. *4 


| Receipt of accelerated and enhanced learning services is not associated with 
program completion. 


The first panel of Exhibit 6-1 presents program completion rates, by receipt of accelerated and 
enhanced learning services; specifically, transfer credits and credits for prior learning in the military, 
apprenticeship programs, or other work experiences. In that panel's first row, the left part shows 
expected and actual completion rates, by receipt of transfer credits (received or did not receive). As 
shown, participants who received transfer credits had an expected completion rate of 43 percent, where 
the expected completion rate is predicted based on their characteristics at program entry. Their actual 
completion rate was slightly higher—44 percent. The excess completion rate was 1 percentage point (not 
shown in the exhibit). Participants who did not receive transfer credits had an expected completion rate 
of 54 percent, slightly higher than their actual completion rate of 53 percent, and thus an excess 
completion rate of —0.2 percentage points (not shown in exhibit). The adjusted difference in program 
completion between those who do or do not receive transfer credits is the difference between the excess 
completion rate for those who received transfer credits and those who did not, which is equal to 

1 percentage point (after rounding). The difference is not statistically significant, suggesting that receipt 
of transfer credits did not have a detectable association with program completion. 


52 The section on receipt of public assistance benefits also reports the adjusted differences of service receipt on poverty 
status. 


53 This section does not examine the effect of the two employment-related services on program completion, since those 
services should not have any direct impact on program completion. 


54 For details on the analytic approach and definitions of the estimates, see the box Analytic Methods and Interpreting 
Adjusted Differences; full details are available in Appendix D. 
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Exhibit 6-1. Adjusted Differences in Program Completion by Service Receipt 


—— Program Completion (%) — Adjusted Difference (%) — 
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Source: Short-term participant follow-up survey 

Note: Sample size is 2,211 survey respondents. For participants who received each service, the expected completion 
rate is denoted with a hollow triangle, and the actual is denoted with a solid triangle. For participants who did not 
receive each service, the expected completion rate is denoted with a hollow square, and the actual is denoted with a 
solid square. The adjusted difference is equal to the difference between actual and expected completion rates for 
participants who received each service, minus the difference between actual and expected completion rates for 
participants who did not receive each service. The adjusted differences are denoted with circles; bounds of 

95 percent confidence interval are denoted with squares. Statistically significant adjusted differences are denoted 
with dark purple circles; insignificant adjusted differences are denoted with light purple circles. Expanded results are 
reported in Appendix G. 
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The second row of that first panel shows expected and actual completion rates, by receipt of credit for 
prior learning. The actual completion rate for participants who received credit for prior learning was lower 
than the expected rate by 3 percentage points (43 percent versus 40 percent). Both expected and actual 
completion rates for those who did not receive credit for prior learning were 52 percent. So the adjusted 
difference in program completion between those who do or do not receive credit for prior learning on 
program completion is —3 percentage points, meaning that receiving credits for prior learning appears to 
actually reduce the likelihood of program completion. However, as with transfer credits receipt, this 
difference was not statistically significant. 


Thus, taking the two services of accelerated and enhanced learning together, this particular category of 
services did not appear to have a connection with completion rates. 


| Receipt of academic advising is associated with a decrease in program 
completion. 


The second panel of Exhibit 6-1 presents program completion rates, by utilization of four persistence 
and completion services: career counseling; academic advising; financial aid advising; and training in 
other skills such as study skills, workplace skills, and general life skills. Only academic advising appears to 
be related to program completion, and the relationship is negative. That is, participants who received 
academic advising were /ess likely to complete their program. As shown, about 46 percent of participants 
who received academic advising were expected to complete their program, but only 43 percent actually 
did so, a difference of —3 percentage points. In contrast, those who did not receive academic advising 
outperformed their expected completion rate by about +3 percentage points. The difference between —3 
and +3 percentage points is statistically significant. The most likely explanation for this result is “reverse 
causation;” that is, participants already struggling with their coursework may have been more likely to 
seek out academic advising than participants who were not struggling. 


| Skills practice in a work-like physical environment is associated with an 
increase in program completion. 


The third panel of Exhibit 6-1 presents program completion rates by utilization of three types of work- 
based learning services: skills practice in a virtual work environment, skills practice in a work-like physical 
environment, and in-program work experience. Among these services, only skills practice in a work-like 
physical environment seemed to be related to program completion. Participants who practiced their skills 
in these environments exceed their expected completion rate by almost 4 percentage points whereas 
those who did not underperformed by 16 percentage points. What could account for this difference? One 
possibility is that work-based learning services sharpen the connection between coursework and actual 
employment. Participants who practice skills have a better understanding of how classroom concepts are 
applied to the “real world” and thus are more likely to complete their program. 


Abt Associates Round 4 Early Outcomes Study Report December 2020 j 13 


CHAPTER 6: PARTICIPANT OUTCOMES, 
BY SERVICE RECEIPT 


| The combination of work-based learning and employment-related services is 
associated with a substantially higher completion rate. 


The team also analyzed program completion rates by different combinations of services. As noted above, 
bundling services can produce biased estimates of the separate services in the bundle. For example, if an 
effective service (that is, one found to be statistically significant) is bundled with a service that has no 
effect, the analyses performed up to this point would find both services effective. If the bundle of 
services does not have a larger adjusted difference than any of its services separately, that is a sign that 
all the services are not separately effective and that one of them may be responsible for the effect of the 
others. This does not help identify which service of the group is the effective one, but it does lead to a 
tempered expectation of what can be achieved with a multi-service program. 


To explore this possibility, the team first studied whether there were common service bundles. This 
research identified four clusters of participants who received distinctly different bundles of services.°° 
Exhibit 6-2 and Exhibit 6-3 show how service receipt varies by cluster. Exhibit 6-2 shows the share 
of participants receiving each of the 11 services by cluster. Exhibit 6-3 summarizes receipt of services 
for each of the four services categories. 


e Cluster 1 (blue bars): Mostly classroom instruction with few other supports. Service receipt for 
participants in this cluster is generally low. In fact, for eight of the 11 services, they have the lowest 
rates of receipt (Exhibit 6-2). They also have the lowest service receipt for two of the four services 
categories (Exhibit 6-3). This cluster includes 595 participants, or 27 percent of the total sample. 


e Cluster 2 (orange bars): Work-based learning with persistence and completion services. 
Participants in this cluster received a mix of most of the 11 services. For seven of them 
(Exhibit 6-2) and two of the four services categories (Exhibit 6-3), participants have the highest 
utilization rates. They stand out for their use of persistence and completion services, particularly 
financial aid advising. This cluster includes 626 participants, or 28 percent of the total sample. 


e Cluster 3 (purple bars): Work-based learning with few other supports. Participants in this cluster 
heavily used work-based learning experiences while using other services sparingly. This cluster 
includes 590 participants, or 27 percent of the total sample. 


e Cluster 4 (gray bars): Work-based learning with employment-related services. Participants in this 
cluster used both employment-related services and work-based learning experiences at high rates. 
This cluster includes 400 participants, or 18 percent of the total sample. 


un 


5 Details of the analytic approach used in the cluster analysis is available in Appendix D. 
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Exhibit 6-2. Receipt of Individual Services, by Participant Cluster 


, LLLLLLLLLLL) 
Transfer credits (XNWKWWKNNRRNRNRRRS 
aE 


Credit for prior learning 


: ULELLTL LL J 
Career cou nselin g LT NTA peel la auld aero LoS eect 
SS 


TAMAR mn 
Academic advising AXKKKKRN 


Fi ial aid advisi HOMME 
Inancial ald AGVISING nw 


TRUCHET nun 
fe oki HUCUCCOTUUaE MHOC Er UCOTNUnaeOeenUEHHTOTNN 
Study, workplace, or life skills COUrSE  (AARNNRNRANRANNANNARAARRRRRRRY 


nner UE 
Skills practice in a Virtual Setting ANA nwin WAAR RAR UNS CUURUNENRICUNRNRARRRRRRRRAN! 


; ot ; : HARARE 
Skills practice in a physical environMent KKK 
eT 


LLL 
Offered work study Job OF INterNship AA WAAANANAANARA ARR 


NJ 
S 
& 
& 
& 
S 
Zar 
Z 
Z 
Z 
Z 


Job search or placement assistance Re ecCeTTeTFETENGEOTHNGTEAIUHNEEKYERRLTHRGKUEEKUHG 


Interviewing pradi ce services -ViabebcabileVabiimoiuiie legible 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
Share Receiving Each Service, by Cluster (%) 


mw Cluster 1: Mostly classroom instruction, few other supports 

Ml Cluster 2: Work-based learning with persistence and completion services 
N Cluster 3: Work-based learning, few other supports 

@ Cluster 4: Work-based learning with employment-related services 


Source: Short-term participant follow-up survey 
Note: Sample size is 2,211 survey respondents. 
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Exhibit 6-3. Receipt of Categories of Services, by Participant Cluster 
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Average Rate of Service Receipt in Each Category, by Cluster (%) 


A Cluster 1: Mostly classroom instruction, few other supports 
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@ Cluster 4: Work-based learning with employment-related services 


Source: Short-term participant follow-up survey 

Note: Sample size is 2,211 survey respondents. “Average receipt of services in each category (%)” is the average 
rate of service receipt across all individual services in each services category. For example, the reported percentage 
for the “accelerated and enhanced learning” category is the average rate of service receipt for its two component 
services—transfer credits and credit for prior learning. 


Exhibit 6-4 shows that completion rates varied significantly across these four participant clusters. 
Participants in cluster 4—who received high levels of both work-based learning and employment-related 
services—dramatically outperformed their expected completion rate, with an excess completion rate of 
19 percentage points. Everyone not in cluster 4 underperformed their expected completion rate by 

4 points (not shown). Taking the difference between these yields an adjusted difference of 23 percentage 
points for the receipt of services by cluster 4 as opposed to the mixture of services received by clusters 1 
through 3. That 23 percentage points is a substantial adjusted difference. It is unclear why participants 
who received that particular bundle of work-based learning services and employment-related services 
have such a high completion rate. A possible explanation is that work-based learning and employment- 
related services make the connection between the training program and actual employment clearer. That 
cluster 1, which includes few services above and beyond classroom training, has a negative and 
significant association with program completion lends credence to this theory. 
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Exhibit 6-4. Connections of Different Service Bundles with Program Completion 


Program Completion (%) — Adjusted Difference (%) 
0% 20% 40% 60% 80% -30% -20% -10% 0% 10% 20% 30% 
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with few other supports 45 54 
Cluster 2. 
Work-based learning with TS 7 © = 
persistence & completion services 40 45 
Cluster 3. 
Work-based learning with ( ) 
few other supports = 
Cluster 4. 
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employment-related services 59 78 
KEY Received Service Adjusted Difference with 95% C.I. 
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Source: Short-term participant follow-up survey 

Note: Sample size is 2,211 survey respondents. For participants in each service cluster, the expected completion rate 
is denoted with a hollow triangle, and the actual is denoted with a solid triangle. The adjusted difference is equal to 
the difference between actual and expected completion rates for participants in each service cluster, minus the 
difference between actual and expected completion rates for participants not in that cluster (i.e., those in any of the 
other three clusters). Adjusted differences are denoted with circles; bounds of 95 percent confidence interval are 
denoted with squares. Statistically significant adjusted differences are denoted with dark purple circles; insignificant 
adjusted differences are denoted with light purple circles. Expanded results are reported in Appendix G. 


6.2. TRAINING-RELATED EMPLOYMENT, BY SERVICE RECEIPT 


This section focuses on how service receipt might be connected to success in obtaining training-related 
employment. All statistics in this section pertain only to participants who either finished their required 
classes or left without finishing, and thus excludes participants still enrolled in training at the time of 
follow-up.*© Exhibit 6-5 shows the expected training-related employment rate, actuva/training-related 
employment rate, and adjusted difference for each of the 11 different services. As discussed in prior 
chapters, participants who finished their classes were much more likely than participants who left without 
finishing to have obtained training-related employment, so any variation in this outcome across 
subgroups may be partially due to differences in enrollment status. This chapter does not explore 
variation in training-related employment separately by enrollment status. 


56 As noted in prior chapters, the survey instrument did not ask about training-related employment for those still in their 
initial programs. 
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| Receipt of transfer credits is associated with higher rates of training-related 
employment. 


The first panel of Exhibit 6-5 presents evidence about connections of accelerated and enhanced 

learning services with participants’ training-related employment after completing the program. The results 
indicate that transfer credits receipt increased the rate of training-related employment. Participants who 
received transfer credits had a 7 percentage point increase (difference between actual and expected 
rates) in training-related employment, whereas those who did not receive transfer credits had a 

2 percentage point decrease. The adjusted difference of 8 percentage points is statistically significant. 
There was no detectable association of receipt of credit for prior learning with training-related 
employment. 


| Receipt of financial aid advising is associated with higher rates of training- 
related employment. 


The second panel of Exhibit 6-5 presents evidence about connections of four persistence and 
completion services with training-related employment. Only one, financial aid advising, appears to have a 
significant association with training-related employment, increasing it by 6 percentage points. It is not 
clear why only this service has an association. It is possible that participants who received financial aid 
advising also received better advising about which training would be more likely to lead to training- 
related employment. 


| Receipt of work-based learning is associated with higher rates of training- 
related employment. 


The third panel of Exhibit 6-5 presents evidence of connections of work-based learning with training- 
related employment. All three types of work-based learning opportunities have positive and statistically 
significant associations with participants’ rates of training-related employment after finishing or stopping 
the program. Participation in work-based learning opportunities situated in physical environments had the 
largest adjusted difference: 15 percentage points. 


| Receipt of employment-related services is associated with higher rates of 
training-related employment. 


The fourth panel of Exhibit 6-5 presents evidence of connections of two employment-related services 
with training-related employment: job search or placement assistance and interviewing practice. Both 
service types are associated with positive and statistically significant increases in participants’ rates of 
training-related employment after completing the program. Participants who received each type of 
service were more than 17 percentage points more likely be employed in a job related to their training 
than were participants who did not receive the services. The similarity of the adjusted differences in 
outcomes between those receiving and not receiving each of the two services raises the question of 
whether they work separately or in tandem with each other. 
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Exhibit 6-5. Adjusted Differences in Training-Related Employment by Service Receipt 
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Source: Short-term participant follow-up survey 

Note: Sample size is 1,834 survey respondents who had finished classes or left without finishing by about 15 months 
after program entry. For participants who received each service, the expected training-related employment rate is 
denoted with a hollow triangle, and the actual is denoted with a solid triangle. For participants who did not receive 
each service, the expected training-related employment rate is denoted with a hollow square, and the actual is 
denoted with a solid square. The adjusted difference is equal to the difference between actual and expected 
employment rates for participants who received each service, minus the difference between actual and expected 
employment rates for participants who did not receive each service. The adjusted differences are denoted with 
circles; bounds of 95 percent confidence interval are denoted with squares. Statistically significant adjusted 


differences are denoted with dark purple circles; insignificant adjusted differences are denoted with light purple 
circles. Expanded results are reported in Appendix G. 
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| The combination of work-based learning and employment-related services is 
associated with higher rates of training-related employment. 


Exhibit 6-6 examines the effect of receiving different bundles of services on the likelihood of training- 
related employment.*” The bundle provided to cluster 1—mostly classroom instruction with few other 
supports—had a significant and negative association with training-related employment (as opposed to the 
bundle of services provided to clusters 2 through 4). Meantime, the bundle of services provided to 
cluster 4—which had high rates of both work-based learning and employment-related services—had a 
significant and positive association with training-related employment (as opposed to the bundle of 
services provided to clusters 1 through 3). This is similar to the results for program completion, which 
found that the combination of work-based learning and employment-related services is associated with a 
higher rate of program completion. 


Exhibit 6-6. Connections of Different Service Bundles with Training-Related Employment 


Training-related Employment (%) Adjusted Difference (%) 
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Source: Short-term participant follow-up survey 

Note: Sample size is 1,834 survey respondents who had finished classes or left without finishing by about 15 months 
after program entry. For participants in each service cluster, the expected training-related employment rate is 
denoted with a hollow triangle, and the actual is denoted with a solid triangle. The adjusted difference is equal to the 
difference between actual and expected training-related employment rates for participants in each service cluster, 
minus the difference between actual and expected training-related employment rates for participants not in that 
cluster (i.e., those in any of the other three clusters). The adjusted differences are denoted with circles; bounds of 
95 percent confidence interval are denoted with squares. Statistically significant adjusted differences are denoted 
with dark purple circles; insignificant adjusted differences are denoted with light purple circles. Expanded results are 
reported in Appendix G. 


57 For the definitions of each of the four clusters, see the discussion introducing Exhibits 6-2 and 6-3 in Section 6.1. 
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6.3. CHANGE IN EARNINGS, BY SERVICE RECEIPT 


This section reports connections of service receipt with change in earnings between the third quarter 
before program entry and the fifth quarter after program entry. Similar to the previous section, the 
sample is restricted to those not still enrolled in their initial training program at survey follow-up. 
Exhibit 6-7 shows the association of each of the 11 services with change in earnings for this set of 
participants. 


| Receipt of transfer credits is associated with a larger increase in earnings. 


The first panel of Exhibit 6-7 shows that among participants who were not still enrolled in training at 
follow-up, those who received transfer credits outperformed their expected change in earnings by 
+$1,104 ($3,538 actual versus $2,434 expected). Participants who did not receive transfer credits earned 
$122 less than their expected change in earnings at program entry ($1,815 actual versus $1,937 
expected). Thus, transfer credits have an estimated statistically significant adjusted difference of $1,226. 
Receipt of credit for prior learning did not have an association with earnings growth. 


| Receipt of persistence and completion services is not associated with the 
change in earnings. 


The second panel of Exhibit 6-7 shows the estimated associations of four persistence and completion 
services with change in earnings. None of the services shows a significant adjusted difference. Although 
the prior section showed a positive adjusted difference of financial aid advising on training-related 
employment, that service is not significantly associated with change in earnings. 


| Participation in virtual work-based learning opportunities is associated with a 
larger increase in earnings. 


The third panel of Exhibit 6-7 shows the association of work-based learning services with change in 
earnings. Participation in virtual work-based learning opportunities is associated with a statistically 
significant increase in earnings of +$947. There was no association for participation in work-like physical 
environments or opportunities for direct work experience. 


| Receipt of employment-related services was not associated with the change in 
earnings. 


The fourth panel Exhibit 6-7 shows the association of two employment-related services with change in 
earnings. Neither job search or placement assistance nor interviewing practice was associated with the 
change in earnings. 
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Exhibit 6-7. Connections of Service Receipt with Change in Earnings 
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Source: Short-term participant follow-up survey 

Note: Sample size is 1,571 survey respondents who had finished classes or left without finishing by about 15 months 
after program entry and had sufficient information at program entry for matching to NDNH wage records. For 
participants who received each service, the expected change in earnings is denoted with a hollow triangle, and the 
actual is denoted with a solid triangle. For participants who did not receive each service, the expected change in 
earnings is denoted with a hollow square, and the actual is denoted with a solid square. The adjusted difference is 
equal to the difference between actual and expected change in earnings for participants who received each service, 
minus the difference between actual and expected change in earnings for participants who did not receive each 
service. The adjusted differences are denoted with circles; bounds of 95 percent confidence interval are denoted with 
squares. Statistically significant adjusted differences are denoted with dark purple circles; insignificant adjusted 
differences are denoted with light purple circles. Expanded results are reported in Appendix G. 
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| No bundle of service receipt is associated with the change in earnings. 


Exhibit 6-8 examines the effect of receiving the four bundles of services on change in earnings.** There 
were no significant adjusted differences for any of the four clusters of participants. This differs from the 
results for program completion and training-related employment, on which the services that cluster 4 
received (high rates of both work-based learning and employment-related services) had large and 
positive adjusted differences. 


Taken together, there is little evidence that receipt of various bundles of services is associated with 
positive change in earnings. Only receipt of transfer credits and skills practice in a virtual environment 
were associated with earnings increases. 


Exhibit 6-8. Connections of Different Service Bundles with Change in Earnings 


Change in Earnings ($) ——— Adjusted Difference ($) ——— 
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Source: Short-term participant follow-up survey 

Note: Sample size is 1,571 survey respondents who had finished classes or left without finishing by about 15 months 
after program entry and had sufficient information at program entry for matching to NDNH wage records. For 
participants in each service cluster, the expected change in earnings is denoted with a hollow triangle, and the actual 
is denoted with a solid triangle. The adjusted difference is equal to the difference between actual and expected 
change in earnings for participants in each service cluster, minus the difference between actual and expected change 
in earnings for participants not in that cluster (i.e., those in any of the other three clusters). The adjusted differences 
are denoted with circles; bounds of 95 percent confidence interval are denoted with squares. Statistically significant 
adjusted differences are denoted with dark purple circles; insignificant adjusted differences are denoted with light 
purple circles. Expanded results are reported in Appendix G. 


58 For the definitions of each of the four clusters, see the discussion introducing Exhibits 6-2 and 6-3 in Section 6.1. 
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6.4. PUBLIC ASSISTANCE BENEFIT RECEIPT, BY SERVICE 
RECEIPT 


This section examines how service receipt might be connected to receipt of public assistance benefits. 
The TAACCCT theory of change (Chapter 2) suggests that completing the offered trainings should lead to 
higher earnings and indirectly to lower receipt of public assistance benefits. The analysis includes all 
short-term participant follow-up survey respondents. 


The findings for public assistance benefit receipt are rather surprising: receipt of many TAACCCT services 
is associated with statistically significant increases in public assistance benefit receipt (Exhibit 6-9). This 
result does not appear to be due to challenges in the study predicting public assistance benefit receipt; in 
fact, the research team predicted benefit receipt at follow-up from variables measured at program entry 
well.°? Moreover, an exploratory analysis of connections of service receipt with poverty level found no 
significant association, suggesting that the apparent association with public assistance benefit receipt is 
not a response to any increase in poverty among participants. © 


| Receipt of credit for prior learning is associated with higher receipt of public 
assistance benefits. 


The first panel of Exhibit 6-9 suggests that one accelerated and enhanced learning service increased 
participants’ public assistance benefits receipt. Participants who received credit for prior learning were 
significantly more likely to receive public assistance at follow-up than participants who did not, with an 
estimated adjusted difference of 11 percentage points. There was no significant difference in public 
assistance benefit receipt based on receipt of transfer credits. 


| Receipt of career counseling, academic advising, and financial aid are all 
associated with higher receipt of public assistance benefits. 


The second panel of Exhibit 6-9 examines the possible connections of persistence and completion 
services with public assistance benefit receipt. Participants who received career counseling, academic 
advising, or financial aid advising were more likely to receive public assistance benefits compared to the 
expected rates at program entry. Participants who did not receive those services had lower rates of public 
assistance benefit receipt compared to the expected rates at program entry. Those adjusted differences 
are statistically significant and range from 5 to 7 percentage points. Given the lack of an association of 
career counseling, academic advising, or financial advising with poverty levels, as noted above, it seems 
plausible that these services are bundled with services that inform participants of their eligibility for public 
assistance programs. 


59 The R-squared coefficient of the team’s prediction model was 19 percent, a very high level for a binary outcome. Some 
of the important predictors at program entry of receipt of public assistance benefits at follow-up included SNAP receipt, 
TRA receipt, living rent free, prior education, family income, employment status, expected work hours during training, 
and citizenship. Program duration also affected receipt—participants in longer programs were more likely to be receiving 
public assistance at the time of the short-term follow-up survey. The team controlled on program duration in the same 
manner as those other characteristics. 


60 See Appendix G for detailed tables on the adjusted differences in public assistance benefit receipt and poverty by service 
receipt. 
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Exhibit 6-9. Adjusted Differences in Receipt of Public Assistance Benefits by Service Receipt 
- Adjusted Difference (%) — 
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Source: Short-term participant follow-up survey 

Note: Sample size is 2,211 survey respondents. For participants who received each service, the expected receipt of 
public assistance benefits is denoted with a hollow triangle, and the actual is denoted with a solid triangle. For 
participants who did not receive each service, the expected receipt of public assistance benefits is denoted with a 
hollow square, and the actual is denoted with a solid square. The adjusted difference is equal to the difference 
between actual and expected receipt of public assistance benefits for participants who received each service, minus 
the difference between actual and expected receipt of public assistance benefits for participants who did not receive 
each service. The adjusted differences are denoted with circles; bounds of 95 percent confidence interval are 
denoted with squares. Statistically significant adjusted differences are denoted with dark purple circles; insignificant 
adjusted differences are denoted with light purple circles. Expanded results are reported in Appendix G. 
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| Being offered a work study job or internship is associated with higher receipt 
of public assistance benefits. 


The third panel of Exhibit 6-9 shows that participants who had the opportunity to participate in a work- 
study job or internship while in training were more likely to receive public assistance benefits at follow-up 
than at program entry by +2 percentage points. In contrast, participants who did not interact with 
employers experienced a reduction in public assistance benefits receipt by —2 percentage points. The 
estimated adjusted difference of 4 percentage points is statistically significant. It is unclear why there is 
an impact of this service on public assistance benefit receipt. 


| Receipt of employment-related services is not associated with higher receipt 
of public assistance benefits. 


The fourth panel of Exhibit 6-9 shows that there is no significant relationship between receipt of 
employment-related services and receipt of public assistance benefits. Neither expected nor actual 
participation rates varied much by receipt of either job search or placement assistance or interviewing 
practice. 


| Receipt of few services across all four services categories is associated with 
lower rates of public assistance benefit receipt. 


Exhibit 6-10 examines possible connections of different bundles of services with receipt of public 
assistance benefits.°! As shown, public assistance benefit receipt was lower than the expected rate only 
for participants in cluster 1. Recall that the participants in cluster 1 received occupational training and 
little else. Being in cluster 1 reduced public assistance benefit receipt by —5 percentage points. In 
contrast, participants in cluster 2, where participants received an array of work-based learning 
opportunities and persistence and completion services, were +8 percentage points more likely to receive 
public assistance benefits at follow-up. Participation in the other two clusters did not appear to be 
associated with public assistance benefits receipt. 


61 For the definitions of each of the four clusters, see the discussion introducing Exhibits 6-2 and 6-3 in Section 6.1. 
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Exhibit 6-10. Connections of Different Service Bundles with Receipt of Public Assistance 
Benefits 


Public Assistance Receipt (%) |——— Adjusted Difference (%) 
0% 10% 20% 30% -10% 0% 10% 
Cluster 1. 
Mostly classroom instruction yee 
with few other supports 16 20 »_ @ : 
Cluster 2. 
Work-based learning with A A 8] 
persistence & completion services 90 26 
Cluster 3. 
Work-based learning with AA ( 2) 
few other supports 16 18 
Cluster 4. 
Work-based learning with A C 1) 
employment-related services 20 21 
KEY Received Service Adjusted Difference with 95% C.I. 
Public Assistance 
Receipt Expected A Actual A Insignificant =(_)}-= Significant = @-= 


Source: Short-term participant follow-up survey 

Note: Sample size is 2,211 survey respondents. For participants in each service cluster, the expected receipt of 
benefits is denoted with a hollow triangle, and the actual is denoted with a solid triangle. The adjusted difference is 
equal to the difference between actual and expected receipt of public assistance benefits for participants in each 
service cluster, minus the difference between actual and expected receipt of public assistance benefits for 
participants not in that cluster (i.e., those in any of the other three clusters). The adjusted differences are denoted 
with circles; bounds of 95 percent confidence interval are denoted with squares. Statistically significant adjusted 
differences are denoted with dark purple circles; insignificant adjusted differences are denoted with light purple 
circles. Expanded results are reported in Appendix G. 
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i. Projected Participant Outcomes 
for Future Cohorts, by Program 


Chapters 4 and 5 focused on outcomes for the full sample of study 
participants as well as for subgroups of participants. Chapter 6 focused 
on the ways in which supportive services can affect participant 
outcomes. This chapter shifts the focus to outcomes for the 

34 programs. It addresses the following research question: 


Exhibits in this chapter show the rankings 
of the 34 programs relative to one 
another. When interpreting these 
findings, it is important to keep in mind 
the rankings are not corrected for 
participant characteristics at program 
entry or for local labor market conditions 
over the follow-up period. A high score 
could indicate the program attracted 
better prepared participants, did a better 
job of training participants, or trained 
participants for occupations in high 
demand at the time of follow-up. Thus, 
the rankings are most likely due to a 
complex set of factors. As such, this 
chapter primarily presents the rankings 
with limited speculation about causes. 


Second, there are many ties and near- 
ties in the rankings, and some of the 
program sample sizes are very small. 
Therefore, it is not clear how the 
rankings would look if all programs scaled 
up. Because of the small sample sizes at 
most programs, the estimates presented 
are not simple averages of outcomes for 
the study period, but are projections of 
outcomes that future cohorts might 


Warning 


The analysis methods in 
this chapter make strong 


assumptions. Reader 
caution is advised. 


Key Takeaways: Participant 
Outcomes at Individual Programs on 


IT programs at Ivy Tech Community College of 
Indiana have low completion rates and low rates 
of training-related employment but strong 
earnings growth once students finish or leave the 
program. 


The shortest programs like the Commercial 
Driver's License and Forklift program at Cincinnati 
rs} r= 1K Maxoln) al (or=] r=) a (elm Golan lanielalinva @e)|(=1e(omr-lavemaal=y 
Core Plus program at Delgado Community College 
have high completion rates but low earnings 
growth. 


The LPN program at Washburn University of 
Topeka has a high completion rate, the highest 
rate of training-related employment and strongest 
earnings growth. 


The Mechatronics and Machining programs at 
South Central College have low completion rates 
but average or higher rates of training-related 
employment coupled with strong earnings growth. 


Welding programs at several grantees have above 
average earnings growth. 


Programs at Chafee Community College tend to 
have below average earnings growth. 


experience at these programs assuming program features, the local labor market, and the participant 
profile are the same as in 2017-2019. The methods used for this report are standard in fields where 
ranking of individual entities is important, such as player selection in professional sports, hospital quality, 
and allocation of need-driven federal education aid. These methods discount exceptional performance 
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(both good and bad) and assume that future performance for these entities will be closer to average, 
particularly if measured performance is based on small sample sizes. See Appendix E for a detailed 
description of these methods as well as comparisons of these projections with simple local mean 
outcomes. 


Interpreting 
Exhibits in Chapter 7 


In this chapter's exhibits, each result is shown using a circle, a thick red bar, and a thin 
green bar. 


The circle represents the point estimate—that is, the average predicted outcome for 
future participant cohorts in that particular program. An example is: 


About a year after program entry, 52 percent of future participants at the machining 
program at South Central College who are not still enrolled in their training program are 
predicted to be employed in jobs closely related to that training. 


This is not an estimate of training-related employment for the participants in the study 
sample who responded to the follow-up survey. Rather, it is an estimate of the inherent rate 
that future cohorts of participants with demographic and economic characteristics similar to 
this study’s participants will experience (assuming program administration remains constant 
and the job market conditions remain similar). The Bayesian estimation procedure discounts 
both extremely high and extremely low rates across the program, under the assumption that 
these are chance blips, unlikely to be repeated, and it “pulls” these extreme estimates toward 
the center. 


The thin blue bar shows what is possible given the local program’s design, the 
characteristics of the participants enrolled, its faculty, and the local economic conditions of 
2017-2019. Continuing with the South Central College example, 


The possible range of completers employed in jobs closely related to their training is 
15 percent to 89. There is a 99.9 percent chance that the true training-related 
employment rate is within this range. 


The thick orange bar shows the likely range and is akin to the confidence interval 
graphics used in Chapters 5 and 6. This bar is the 80 percent credible interval. For the South 
=T p=] Oo) (=re (= aat-revalial ave wm o)ceyele- laa 


The estimate for training-related employment runs from 35 to 69 percent. This is the 
shortest interval that has an 80 percent probability of containing the true rate for future 
cohorts of participants under the same assumptions noted above. Nonetheless, there is a 
10 percent probability that the true rate is smaller than 35 percent, and a 10 percent 
probability that the true rate is larger than 69 percent. 


For more details on the Bayesian method see Appendix E. 


This chapter, as with other chapters, includes graphics that present both point estimates and intervals 
around these point estimates that indicate the precision of the estimate. However, the statistical 
approach to constructing these intervals is very different than in prior chapters. The intervals in prior 
chapters were confidence intervals; the intervals in this chapter are “credible intervals.” Confidence 
intervals reflect hypothetical repetitions of the data collection; in Chapters 5 and 6, they were constructed 
such that if the data collection were repeated many times, in 95 percent of cases the computed 
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confidence interval would contain the true value of the outcome. In contrast, credible intervals take into 
account a prior “belief” about the true outcome level.® A credible interval on the best estimate for a 
specific program reflects the best judgment of authors about the amount of uncertainty in the predicted 
average experience of future cohorts of participants in a program, assuming no changes in program 
design, staff quality at the program, average skills and interests of new cohorts admitted to the program, 
and local economic conditions for graduates. Both 80 and 99.9 percent credible intervals are shown. 


Although the research team generally prefers confidence intervals, the technical challenges in producing 
them are much greater for program estimates than for the types of statistics presented in the prior 
chapters, so credible intervals are reported in Chapter 7.° 


7.1. PROGRAM COMPLETION 


Exhibit 7-1 shows program completion for the 34 programs in the outcomes study based on responses 
to the short-term participant follow-up survey. As discussed in earlier chapters, program completion 
means both finishing required classes and earning an associated credential. 


| The commercial driver’s license and forklift programs had the highest rates of 
program completion. 


As shown, the research team projects the commercial driver's license and forklift programs at Cincinnati 
State Technical and Community College to have the highest completion rate (93 percent). The orange 
and blue intervals are tight, indicating that there is an 80 percent probability that the true completion 
rate is between 90 and 96 percent, and a 99.9 percent probability that it is between 82 and 99 percent. 
The short average duration of one week for these programs is likely an important driver of the high 
completion rates. 


| Health care programs, some welding programs, and manufacturing programs 
also had high completion rates. 


The Washburn University consortium’s licensed practical nurse program, for example, had an estimated 
completion rate of almost 80 percent, and the 80-percent plausible completion range is 73 to 85 percent. 
The other programs in the Washburn University consortium had completion rates that also were high 
(74 percent). The Chaffey College mechanical craft program had a similar estimated completion rate 

(79 percent), but a larger plausible range, due mostly to a smaller number of program participants. 
Perhaps the career navigators in the Washburn University consortium provided important services that 
assisted participants to persist in and complete their training programs. 


62 The philosophy behind confidence intervals is known as “frequentist inference.” The one behind credible intervals is 
known as “Bayesian inference.” 


63 There are different traditions with each philosophy about how much confidence/credibility should be built into the 
intervals. Confidence intervals are usually built to contain 95 percent confidence, and that is the tradition the research 
team followed in prior chapters. The software used for credible intervals has defaults of 80 and 99.9 percent credibility, 
which the research team used in this chapter. 
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Exhibit 7-1. Projected Program Completion Rates for Future Cohorts, by Program 


CDL and Forklift (Cincinnati) 

Core Plus (Delgado) 

Licensed Practical Nurse (Washburn) 
Mechanical Craft (Chaffey) 

Emergency Medical Technician (Washburn) 
Welding Technology (Washburn) 

Advanced Manufacturing and Engineering (Chaffey) 
Advanced Manufacturing (Manchester) 
Welding Technology (Manchester) 

Industrial Automation (Chaffey) 

Industrial Maintenance (Chaffey) 

TECH 101 (Delgado) 

TRAMCON Advanced (Miami Dade) 
TRAMCON Basic (Miami Dade) 
Pre-engineering (Chaffey) 

TRAMCON Foundation (Miami Dade) 

Welding (South Central) 

Advanced Welding Bootcamp and Program (Bossier) 
Fast Track to Manufacturing (Bossier) 

Certified Production Technician (South Central) 
Right Skills Now (South Central) 

HVAC (Chaffey) 

Gateway Courses to IT Programs (Ivy Tech) 
Machining (South Central) 

Informatics (Ivy Tech) 

Welding (Chaffey) 

Mechatronics (South Central) 

Network Infrastructure (Ivy Tech) 

Software Development (Ivy Tech) 

Information Technology Support (Ivy Tech) 


Programs 


Cyber Security (Ivy Tech) 
Database Management (Ivy Tech) 
Computer Science (Ivy Tech) 
Server Administration (Ivy Tech) 


0.00 0.25 0.50 0.75 1.00 


Source: Short-term participant follow-up survey 

Note: Sample is 2,211 survey respondents. Program completion means finishing the required classes and earning an 
associated credential. Circle marks projected completion rate for future cohorts at the program. Thin blue line traces 
99.9 percent credible interval. Thick orange line traces 80.0 percent credible interval. 
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| Among the 34 programs, information technology programs had the lowest 
rates of program completion. This was likely due to these programs’ long 
durations. 


At the other end of the spectrum, the Ivy Tech Community College IT-related programs had the lowest 
completion rates, ranging from 2 percent for server administration to 10 percent for network 
infrastructure. The orange bars for the five lowest-ranked programs (all Ivy Tech Community College) 
barely extend beyond the circle marking the point estimate, indicating there is an 80 percent probability 
that each of those programs had very low completion rates within the follow-up period. Even if one 
considers 99.9 percent credible intervals, it is extremely unlikely that four of these programs have 
completion rates greater than 20 percent. 


What might account for the low completion rates of those IT programs at Ivy Tech Community College? 
Its eight programs included in the outcomes study were the longest in the study, with some lasting 
almost two years. As noted in Chapter 2, most follow-up survey interviews occurred between 12 and 

15 months after program entry—too soon for participants to have completed these trainings. The server 
administration program was one of the longer ones (21 months); the network infrastructure program was 
the shortest at the college (14 months), but still considerably longer than other programs in the study. 
The outcomes study also included two short (four months) Ivy Tech Community College gateway classes 
to its IT programs; most of the college’s study participants began in one of these classes. Based on the 
change in earnings impact of all of these IT programs, discussed below in Section 7.3, it appears that 
program completion may not be a requirement for positive earnings gains. In addition to having some of 
the longest programs in the outcomes study, many Ivy Tech Community College classes were offered 
partly or wholly online. Completion rates in online classes tend to be substantially lower than in-person 
Classes. 


7.2. TRAINING-RELATED EMPLOYMENT 


Exhibit 7-2 shows rates of employment in a training-related occupation for participants in the 

34 programs selected for the outcomes study. The analysis is restricted to study participants who were 
no longer enrolled at follow-up because they had either finished the required classes or left without 
finishing. 


| The licensed practical nurse and a welding technology program had the 
highest rates of training-related employment. 


As shown, the Washburn University licensed practical nurse program had the highest rate of training- 
related employment, with an estimated average of 84 percent. Moreover, the plausible ranges for the 
program are relatively small, suggesting that future cohorts would have a similar training-related 
employment rate. 
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Exhibit 7-2. Projected Training-related Employment Rates for Future Cohorts, by Program 


Licensed Practical Nurse (Washburn) 

Welding Technology (Washburn) 

Advanced Manufacturing (Manchester) 
Mechanical Craft (Chaffey) 

Machining (South Central) 

Pre-engineering (Chaffey) 
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Advanced Manufacturing and Engineering (Chaffey) 
Certified Production Technician (South Central) 
Advanced Welding Bootcamp and Program (Bossier) 
Welding (South Central) 

Welding Technology (Manchester) 

Industrial Maintenance (Chaffey) 

Fast Track to Manufacturing (Bossier) 
TRAMCON Basic (Miami Dade) 

Right Skills Now (South Central) 

CDL and Forklift (Cincinnati) 

Mechatronics (South Central) 

TRAMCON Foundation (Miami Dade) 
Emergency Medical Technician (Washburn) 
HVAC (Chaffey) 

TECH 101 (Delgado) 

Welding (Chaffey) 

Gateway Courses to IT Programs (Ivy Tech) 
TRAMCON Advanced (Miami Dade) 

Core Plus (Delgado) 

Network Infrastructure (Ivy Tech) 


Programs 


Server Administration (Ivy Tech) 

Cyber Security (Ivy Tech) 

Informatics (Ivy Tech) 

Software Development (Ivy Tech) 

Database Management (Ivy Tech) 
Information Technology Support (Ivy Tech) 
Computer Science (Ivy Tech) 


0.00 0.25 0.50 0.75 1.00 


Source: Short-term participant follow-up survey 
Note: Sample is 1,834 survey respondents who had finished classes or left without finishing by about 15 months after 
program entry. Circle marks projected completion rate for future cohorts at the program. Thin blue line traces 
99.9 percent credible interval. Thick orange line traces 80.0 percent credible interval. 
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The welding technology program at Washburn University had the next highest training-related 
employment rate (57 percent). Unlike the licensed practical nurse program, the welding technology 
program’s 80-percent plausible range for future cohorts is wide, ranging from 45 to 68 percent. As noted 
above, the welding program had a completion rate of 74 percent, one of the highest observed. It is 
unclear why its training-related employment rate is lower than its completion rate. The implementation 
study reported that welding positions in the gas industry (e.g., repairing pipelines) had been expected to 
be in high demand but were not, affected by a slowdown in the gas and oil industry. Although Washburn 
University encouraged completers to explore other industries in the area or work out of state, it appears 
that many participants did not obtain welding-related jobs. 


| IT programs other than the gateway classes had the lowest training-related 
employment rates. 


The rates of training-related employment for these programs ranged from 6 percent in computer science 
to 12 percent in network infrastructure. The 80 percent credible intervals for these programs range from 
lower bounds of 0 percent (computer science, database management, informatics) to upper bounds of 
22 percent (server administration). The low training-related employment rates are perhaps not surprising, 
given the low program completion rates, possibly due to the longer length of the programs. Given IT 
participants’ higher-than-average earnings, as described in the next section, it is possible that these 
participants were working in the IT field, but in an occupation that was only somewhat related, rather 
than closely related to their training program. 


| Only three other programs had estimated training-related employment rates 
higher than 50 percent. 


These included the Manchester Community College consortium’s advanced manufacturing program, the 
Chaffey College consortium’s mechanical craft program, and the South Central College consortium’s 
machining program. The completion rate for these programs ranged from a high of 79 percent for 
mechanical craft to about 66 percent for advanced manufacturing to a low of 17 percent for machining. 
Looking more closely at the South Central College machining program, its training-related employment 
rate is higher than its completion rate. It is possible that participants earned in-demand skills in the 
program and were hired prior to completing it. 


| The programs with the highest completion rates had only average estimated 
training-related employment rates. 


The estimated training-related employment rate for the Cincinnati State Technical and Community 
College programs—27 percent—could reflect that few jobs were available in the area. Or the programs 
may not have prepared participants for the jobs in demand locally. The Delgado Community College Core 
Plus program resulted in a National Center for Construction Education & Research Core credential and the 
U.S. Occupational Safety and Health Administration (OSHA)’s OSHA 10 credential, which may not have 
been aligned with jobs in the local economy. 
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7.3. CHANGE IN EARNINGS 


Exhibit 7-3 shows change in earnings for study participants no longer enrolled in their program. The 
change in earnings measure assesses a two-year period: three quarters prior to program entry plus five 
quarters after program entry. 


Exhibit 7-3. Projected Two-Year Changes in Earnings for Future Cohorts, by Program 
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Software Development (Ivy Tech) 

Welding (South Central) 

Welding Technology (Manchester) 
Information Technology Support (Ivy Tech) 
Mechanical Craft (Chaffey) 

Right Skills Now (South Central) 

Advanced Manufacturing (Manchester) 
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Certified Production Technician (South Central) 
Gateway Courses to IT Programs (Ivy Tech) 
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Emergency Medical Technician (Washburn) 
Advanced Manufacturing and Engineering (Chaffey) 
HVAC (Chaffey) 

TRAMCON Advanced (Miami Dade) 

Advanced Welding Bootcamp and Program (Bossier) 
Fast Track to Manufacturing (Bossier) 
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Source: National Directory of New Hires 

Note: Sample size is 1,571 survey respondents who had finished classes or left without finishing by about 15 months 
after program entry and had sufficient information at program entry for matching to NDNH wage records. Study 
period covers the three quarters before program entry through the five quarters after it. Circle marks projected 
change in earnings for future cohorts at the program. Thin blue line traces 99.9 percent credible interval. Thick 
orange line traces 80.0 percent credible interval. 
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| Ofthe 11 programs with the most positive change in quarterly earnings, seven 
were Ivy Tech Community College IT programs. 


The Ivy Tech Community College programs increased earnings between $3,040 and $5,066 per quarter 
depending on the program. This finding was surprising, given their low levels of program completion and 
training-related employment. It suggests that program participants are finding employment, possibly in 
the IT industry, but not specific to their training. Two programs at the South Central College consortium 
and two at the Washburn University consortium also produced very positive changes in earnings. 
Program participants in the mechatronics program at South Central College could expect earnings to 
increase $4,380 per quarter, and those in the machining program, $3,612. Program participants in the 
welding technology program at the Washburn University consortium could expect an increase in earnings 
of $3,420 per quarter, and those in the licensed practical nurse program, $3,239. 


| Among participants with the lowest estimated change in earnings were those 
who attended construction programs, certain manufacturing programs, and 
the commercial driver’s license and forklift programs. 


The Core Plus program at Delgado Community College produced the lowest change in earning, $422 per 
quarter. As noted above, its credentials may not have prepared recipients for jobs in the local economy. 
Delgado Community College's TECH 101 program had a positive change in earnings of $856 per quarter. 
This program also resulted in National Center for Construction Education & Research Core and OSHA 

10 credentials. Again, it is possible the credentials had limited labor market value in the New Orleans 
region. The commercial driver’s license and forklift program at Cincinnati State Technical and Community 
College each yielded $665 per quarter. 


The Miami Dade College Training for Manufactured Construction (TRAMCON) foundation and basic 
programs were associated with positive changes in earnings of $683 and $1,088 per quarter, 
respectively. The foundation program was the first of a series of four TRAMCON programs, each 
associated with a credential. It is possible that the credential from the first program in the series was not 
sufficient for employment in the manufactured building industry. During the research team’s site visit, 
program staff reported that there were fewer jobs in manufactured construction than initially expected. 
At the time of grant application, the Miami Dade College consortium identified an increasing demand for 
homes and buildings that used manufactured construction and a shortage of trained workers. However, 
that demand for manufactured construction did not materialize for a variety of local reasons. These 
market conditions appear to have affected the earnings growth of TRAMCON participants. 


Finally, participants in the three Chaffey College programs (pre-engineering, industrial maintenance, and 
industrial automation) had below average positive changes in earnings, ranging from about $1,011 to 
$1,270 per quarter. 


7.4. PUBLIC ASSISTANCE BENEFIT RECEIPT 


Finally, the research team explored public assistance benefits receipt among program participants at the 
time of the short-term follow-up survey—including participants still enrolled in their program. 
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| Asmall percentage of participants received public assistance benefits at the 
time of the follow-up survey. 


As shown in Exhibit 7-4, the Chaffey College welding program participants were outliers. An estimated 
38 percent received public assistance benefits at the time of the survey; the next largest estimate is 

26 percent at the Delgado Community College Core Plus program, which, as described above, had a low 
training-related employment rate. Welding had the lowest completion rate among Chaffey College’s 
programs (12 percent), although it was its longest program (18 months). 


Exhibit 7-4. Projected Public Assistance Benefits Receipt for Future Cohorts, by Program 
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TECH 101 (Delgado) 

Server Administration (Ivy Tech) 
Network Infrastructure (Ivy Tech) 
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TRAMCON Advanced (Miami Dade) 
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Welding Technology (Manchester) 
Licensed Practical Nurse (Washburn) 


Software Development (Ivy Tech) 
Database Management (Ivy Tech) 
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Machining (South Central) 
Informatics (Ivy Tech) 


Welding Technology (Washburn) 


Mechanical Craft (Chaffey) 


0.0 0.2 0.4 0.6 


Source: Short-term participant follow-up survey 
Note: Sample is 2,211 survey respondents. Circle marks projected completion rate for future cohorts at the program. 
Thin blue line traces 99.9 percent credible interval. Thick orange line traces 80.0 percent credible interval. 
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At the other end of the spectrum, two Washburn University programs (emergency medical technician and 
welding technology) and two Chaffey College programs (mechanical craft and advanced manufacturing 
and engineering) had the lowest estimated rates of public assistance benefit receipt. Both Washburn 
University programs had high program completion rates, but mixed training-related employment rates, 
with emergency medical technician program participants having the lower. 


7.5. SUMMARY OF RANKINGS ACROSS THE FOUR OUTCOMES 


Exhibit 7-5 summarizes the outcomes by program by categorizing the programs as those that scored in 
the top, middle, and bottom third of the desired direction for that outcome. Programs ranking in the most 
favorable third are shaded in purple, the middle third in orange, and the least favorable third in gray. The 
table illustrates patterns in program rankings across outcomes. For example, the models predict that 
future cohorts of students at the Welding Technology program at Washburn University will have high 
program completion rates, high rates of success at finding training-related employment, high levels of 
two-year earnings growth, and high levels of success in avoiding the use of public assistance benefits. 


| Few programs ranked high across all four outcomes. 


In general, program rankings varied by the outcome in question. Only one program—welding technology 
(Washburn University consortium)—scored in the top third across all four outcomes. Four programs— 
mechanical craft (Chaffey College), licensed practical nurse (Washburn University consortium), Advanced 
Manufacturing and Engineering (Chaffey College), and machining (South Central College) ranked in the 
top third on three of the four outcomes. 


| Seven programs that ranked low for program completion and training-related 
employment were nonetheless in the top third for change in earnings. 


All seven of these programs were in the information technology sector and offered at Ivy Tech 
Community College. The college had some of the longest programs in the outcomes study, lasting 
between 14 and 24 months. Because earnings are measured about 15 months after program entry, many 
participants had not yet had an opportunity to complete their program. Information technology jobs have 
high earnings potential.®* It is possible that participants who completed their program secured high- 
paying jobs, those who did not complete their program were nonetheless hired at well-paying jobs, and 
those still enrolled obtained jobs while continuing their training. This possibility still begs the question of 
why the college’s participants’ rate of training-related employment was low as compared to other 
programs. Perhaps even if its participants did not secure a job in the particular information technology 
field of their training, they were still able to obtain well-paying jobs in the general sector. For example, 
someone training for a computer science credential may have been able to obtain a job at a corporate 
information technology helpdesk. 


&  https://www.blis.gov/ooh/computer-and-information-technology/home.htm 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1 98 


CHAPTER 7: PARTICIPANT OUTCOMES, BY PROGRAM 


Exhibit 7-5. Summary of Program Rankings, by Outcome 


Welding Technology (Washburn) 


Mechanical Craft (Chaffey) 


Licensed Practical Nurse (Washburn) 
Advanced Manufacturing and Engineering (Chaffey) 


Machining (South Central) 


Emergency Medical Technician (Washburn) 


Welding (South Central) 


Welding Technology (Manchester) 


Advanced Manufacturing (Manchester) 


Certified Production Technician (South Central) 


Mechatronics (South Central) 


Advanced Welding Bootcamp and Program (Bossier) 


Informatics (Ivy Tech) 


Gateway Courses to IT Programs (Ivy Tech) 


Industrial Maintenance (Chaffey) 


TRAMCON Basic (Miami Dade) 


Industrial Automation (Chaffey) 


Right Skills Now (South Central) 


Network Infrastructure (Ivy Tech) 


TRAMCON Advanced (Miami Dade) 


CDL and Forklift (Cincinnati) 


Pre-engineering (Chaffey) 


Software Development (Ivy Tech) 


Server Administration (Ivy Tech) 


Fast Track to Manufacturing (Bossier) 


Database Management (Ivy Tech) 


TECH 101 (Delgado) 


Information Technology Support (Ivy Tech) 


Cyber Security (Ivy Tech) 


HVAC (Chaffey) 


TRAMCON Foundation (Miami Dade) 


Computer Science (Ivy Tech) 


Core Plus (Delgado) 


Welding (Chaffey) 


KEY Most favorable 
Average 
Least favorable 

Abt Associates 


Round 4 Early Outcomes Study Report 


Note: Exhibit is sorted by earnings growth, sum of all four ratings, then training-related employment. 
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| Four programs that ranked high for program completion and training-related 
employment were in the middle or bottom third for change in earnings. 


These programs were mechanical craft, advanced manufacturing and engineering, and industrial 
automation at the Chaffey College consortium; and advanced manufacturing and welding technology at 
the Manchester Community College consortium. It is notable that all four programs are in the 
manufacturing and welding sectors. In general, participants in these programs experienced earnings 
increases, but the programs were rated in the middle (rather than top) third of programs in the change in 
earnings outcome. Jobs in the advanced manufacturing sector do not demand wages as high as those in 
sectors such as information technology. Whereas the median annual wage for computer and information 
technology jobs was $86,320 in 2018,° the median wage for machinists and tool and die makers (an 
example occupation in the advanced manufacturing sector) was $44,950. It may be that industry trends 
in wage rates helped Ivy Tech Community College participants to increase their quarterly earnings more 
than did participants in the Chaffey College consortium’s and Manchester Community College 
consortium’s welding and manufacturing programs. It is important to note, however, that the relatively 
short follow-up period for this study limits what can be learned about wage increases. 


| Nine programs that were in the top or middle third for program completion 
and training-related employment were in the bottom third for change in 
earnings. 


These programs were commercial driver's license and forklift (Cincinnati State Technical and Community 
College); TECH 101 (Delgado Community College); advanced welding bootcamp/program and fast track 
to manufacturing (Bossier Parish Community College); pre-engineering, industrial maintenance, and 
industrial automation (Chaffey College consortium); and TRAMCON basic and foundation (Miami Dade 
College consortium). These 9 were among some of the shorter programs in the outcomes study; the 
average program duration for these programs was 3.1 months. Therefore, it stands to reason that 
participants finished these programs at relatively high rates inside the approximately 15-month follow-up 
period. However, such short-term programs appeared to have not yielded much in earnings gains. Again, 
a longer follow-up period would provide more insight into participants’ longer-term change in earnings. 


| Ofthe 11 programs in the top third for training-related employment, 10 were 
in the manufacturing and welding industries. 


These were programs in the Chaffey College, Manchester Community College, South Central College, and 
Washburn University consortiums plus programs at Bossier Parish Community College. Manufacturing and 
welding programs appear to have performed well in helping their participants to secure training-related 
employment. This performance could be due to the specialized technical skills that participants gained 
through practice on equipment at, for example, Chaffey College’s InTech Center and the Miami Dade 
College consortium’s Transformative Learning Centers. It could also be due to job placement assistance 
provided by, for example, job placement specialists in Cincinnati State Technical and Community College’s 
forklift program or the career coach in Bossier Parish Community College's welding programs. 


65 Ibid. 
66 https://www.bls.gov/ooh/production/home.htm 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1] 100 


CHAPTER 8: SUMMARY AND DISCUSSION 


8. Summary and Discussion 


This report explored a wide range of participant outcomes as well as characteristics at program entry for 
participants who enrolled in 34 programs implemented by nine selected TAACCCT Round 4 grantees. The 
34 programs were generally short-term, culminated in a credential, were fully or largely supported with 
grant funds, and had a specific point of intake that facilitated study enrollment. In addition to outcomes 
for the entire sample of 2,767 participants, the report presents outcomes by participant characteristics 
and participant service receipt for four outcomes: program completion, training-related employment, 
change in earnings, and public assistance benefits receipt outcomes. Finally, it describes projected 
outcomes for future participant cohorts by program for these same four key outcomes. 


The TAACCCT capacity-building theory of change made three assumptions: (1) Grant-funded programs 
would target adult learners who were older; had work experience; and were beneficiaries of the Trade 
Readjustment Allowances (TRA) program, veterans, or unemployed adults.°’ (2) Grantees would 
implement a combination of strategies to support accelerated and enhanced learning, persistence and 
completion, and connections to employment. (3) Grantee strategies would produce positive short-term 
participant outcomes related to program completion, employment in the occupation for which they 
trained, change in earnings, and public assistance benefits receipt. 


This chapter explores the extent to which outcomes study findings aligned with the theory of change. It 
then describes possible reasons for unexpected results. 


8.1. HOW FINDINGS ALIGN WITH SHORT-TERM OUTCOMES IN 
THE THEORY OF CHANGE 


The outcomes study explored the characteristics of participants at program entry and reported their 
education, employment, earnings, and public assistance benefits receipt outcomes approximately 

15 months after program entry. It also explored how outcomes varied by subgroups of participants 
defined by demographic and economic characteristics, by service receipt during the program, and by 
grant-funded program. This section considers how these findings align with the TAACCCT theory of 
change. 


| Most participants were adult learners who were older and had work 
experience, which is consistent with the theory of change. 


Almost 60 percent were 25 or older at the time they entered their program. Most had at least some 
postsecondary education (53 percent), most commonly some college but no degree (32 percent). On 
average, participants were unusually well educated as compared to the low-wage worker population 
more generally. More than half (56 percent) were working at the time they entered their program. Thirty 
(30) percent had children in their household. 


Few participants were TRA recipients or veterans. Less than 2 percent of program participants were TRA 
benefit recipients at the time they entered their program, and less than 9 percent were veterans. Grantee 
staff reported during the research team’s site visits that they attempted to recruit both groups. The 


6” TRA benefits are available under TAA. 
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largest challenge was the small number of TRA recipients and veterans in their catchment areas. TRA 
receipt has always been relatively small given the criteria needed to qualify for trade dislocation-related 
job loss. Moreover, TRA receipt was down overall in 2014, when Round 4 grants were awarded. Many 
grantee colleges and their consortium partners were not in the vicinity of military bases. 


| Grantees implemented a range of capacity-building services designed to 
bolster connections to employment for participants; of them, employment- 
related supports were most commonly used by participants. 


The nine grantees implemented career pathways programs or stackable credentials. The majority 
implemented at least one technology-enabled learning strategy, such as online learning. Five grantees 
implemented prior learning assessments or provided credit for prior learning. Grantees used grant funds 
to provide a range of academic supports. Five grantees provided one-on-one nonacademic supports 
through grant-funded staff such as program navigators or recruitment, retention, and completion 
specialists. 


All grantees implemented skills simulations as part of their grant-funded activities; seven offered in- 
program work-based learning strategies, with internships being the most common. Seven provided job 
search services either informally (from instructors) or through dedicated grant-funded positions, including 
career coaches, job placement specialists, and career developers. 


The most commonly received services relate to employment. More than 80 percent of participants 
reported practicing skills in a work-like physical environment, and about half of participants (48 percent) 
reported practicing skills in virtual settings. More than half (57 percent) reported some type of direct 
work experience as part of their program. Forty (40) percent of participants received job search or 
placement assistance. 


Fewer participants received persistence and completion services. Less than half (47 percent) received 
academic advising and 31 percent received financial aid advising. The most commonly received 
persistence and completion strategies not related to advising were classes in other skills such as acting 
professionally (29 percent), critical thinking and problem solving (27 percent), communicating well 

(26 percent), and working in groups (25 percent). The least common persistence and completion 
strategies received included classes in handling parenting and other family responsibilities (8 percent) 
and managing money and personal finances (10 percent). 


| Most participants completed their training programs. Program completion 
was higher for older participants and those with prior college experience. 


Fifty-nine (59) percent of participants finished the required classes for their program; another 17 percent 
were still enrolled in their program at the time of follow-up. The remainder left without finishing their 
classes. About half (51 percent) of participants finished their classes and earned an associated credential. 
The research team considered this group to have completed the program. Participants with some 
postsecondary education completed their program at a higher rate than those without any postsecondary 
education; similarly, participants age 25 and older had higher program completion rates than those age 
24 or younger. 
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| One-third of the participants who finished their required classes or left without 
finishing were employed in a job closely related to their training at follow-up. 


About 33 percent of participants no longer in training were employed in a job closely related to their 
training. Participants age 25 to 34, and those with more recent employment, had more success finding a 
job closely related to their training than did younger participants with less recent employment experience. 
Participants previously employed in manufacturing had higher rates of training-related employment than 
those previously employed in other industries. This suggests that experienced workers with a stronger 
connection to the labor market—particularly those who previously worked in manufacturing—were more 
able to benefit from grant-funded training and services to find related employment. 


| Quarterly earnings increased by about $2,300 between the third quarter prior 
to program entry and the fifth quarter following program entry. 


Many participants experienced substantial earnings growth: earnings per quarter rose from $4,900 three 
quarters prior to program entry to $7,200 five quarters after program entry. This aligns with the theory of 
change, which predicted that grant-funded training would accompany substantial positive change in 
earnings; however, earnings gains were actually larger for those who left their classes without finishing 
($2,715) than for those who finished classes ($2,224), so it is not clear that the program itself was 
associated with the increase in earnings. In addition, more than one in five participants (22 percent) had 
zero earnings five quarters after program entry, suggesting that at least a portion of the sample was 
unable to find employment. 


Positive change in earnings varied by participant characteristics. Younger participants—especially those 
employed at program entry or who had not worked in the past year—experienced the largest earnings 
gains. Participants enrolled in longer programs had significantly larger earnings gains than those enrolled 
in shorter programs. TRA and SNAP recipients had smaller earnings increases than non-recipients. 


| Asmall share of participants were receiving public assistance benefits. Most 
participants who received SNAP or TANF benefits at program entry were still 
receiving those benefits at follow-up. 


At the time of program entry, 11 percent of participants were receiving SNAP benefits, and just 2 percent 
were receiving TANF. These rates were unchanged by the short-term participant follow-up survey. Most 
participants who were receiving SNAP or TANF at program entry were still receiving public assistance 
benefits at follow-up. Older participants and those with long periods without employment had higher 
rates of public assistance benefit receipt. 


8.2. WHAT DO THE FINDINGS SUGGEST FOR FUTURE 
PROGRAMS? 


The outcomes study findings aligned with the theory of change in some ways (program completion, 
higher post-program earnings) but not in others (training-related employment, receipt of public 
assistance benefits). Grantees implemented a range of services but participation varied. 


The outcomes study found that particular services were associated with positive outcomes. Specifically, 
the findings suggest the following: 
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| Program completion might be further improved through more opportunities 
for work-based learning opportunities, such as skills practice in work-like 
physical environments. 


These opportunities were already a very common service; as noted above, more than 80 percent of 
participants received these services. Those who did not receive this service had a lower program 
completion rate, less success finding training-related employment, and a less positive change in earnings, 
even after correcting for baseline differences between those who did or did not receive the service. 


The programs with the highest completion rates—Cincinnati State Technical and Community College’s 
commercial driver’s license and forklift programs; Delgado Community College’s Core Plus program; 
Washburn University consortium’s licensed practical nurse program; and South Central College 
consortium’s mechanical craft program—have work-based learning as a central component. 


| Specific activities might bolster the prevalence of training-related 
employment. 


The analyses found that work-based learning was associated with training-related employment. The 
program with the highest rate of training-related employment, the Washburn University consortium’s 
licensed practical nurse program, used a state-of-the-art simulated learning center where participants 
could practice their diagnostic and treatment skills in a “real life” setting. 


The analyses also suggest that training-related employment might be boosted through a number of other 
services that were received by a minority of participants. These include transfer credits (only 21 percent 
of participants received any), more financial aid advising (32 percent received any), and employment- 
related services (40 percent received job search or placement assistance and 25 percent got interviewing 
practice). 


| Two services were associated with short-term earnings growth. 


Transfer credits and work-based learning seemed to increase short-term earnings for participants who 
received these services. As noted above, the programs with the most positive change in earnings were 
those that incorporated skills practice. These include Ivy Tech Community College's IT programs, the 
Washburn University consortium’s welding program, the Manchester Community College’s welding 
program, and the South Central College’s mechatronics program. The modest earnings growth may be 
due to the short follow-up period. 


| The analyses did not find any promising approaches to reducing public 
assistance benefit receipt. 


About the same proportion of program participants received public assistance benefits at the time of the 
follow-up survey as did a program entry. It appears that some services increased public assistance 
receipt levels. Participants who received career counseling, academic advising, and financial aid advising 
received public assistance benefits at higher rates than those who did not receive such services. As well, 
participants who received credit for prior learning received public assistance benefits at higher rates at 
follow-up than those who did not receive such credit. 


One possible explanation for this finding is that advising services are often bundled and can include 
referrals to support services, including public assistance programs. Some of the programs associated with 
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higher rates of public assistance benefits receipt, such as Core Plus, incorporated advisors who provided 
a range of services to participants. The Core Plus program navigators attempted to meet with all 
participants to create Individual Service Strategies that, among other things, identified barriers to 
participation and needed referrals to additional services. 


| It may be too early to gauge employment, earnings, and public assistance 
benefits receipt outcomes. 


Because the follow-up period was relatively short (about 15 months between program entry and the 
follow-up survey), it is possible that many of the programs’ effects on employment, earnings, and public 
assistance benefits receipt are not yet observable. A longer study time horizon could shed light on 
whether participants still enrolled in their program finish its classes and earn associated credentials. It 
could also reveal whether those who completed their program return to train for the next step on that 
career pathway; more participants move into training-related employment, perhaps after an initial job not 
related to their training; earnings grow more and for most participants; and public assistance benefits 
receipt declines. 
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